:W&Bá¾&Äo.M sxwt6rmsª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0ä“Úš‚" wv2kuj3q2xzp sxwt6rmsŠÝ sxwt6rmsescheuller-uc-san-diego spatop-sweep"ž _wandb‚{}  batch_size‚1024  dropout‚0.23696665082470517  hidden_dim‚64 ( learning_rate‚0.00010175742803218096  num_embedding_layers‚6Bdpfx_blocks_sxwt6rmsbox8or7c7j$dp-spatop-sweepagent-blocks-v2-fb276Š †õÒø…Æš3 (H¨¸hp"3.12.10*0.28.1j linux-x86_648rÚ”; sxwt6rms*  _wandb‚ {"runtime":0}  _runtime‚0ñZ­B`ÒÚ -Linux-6.8.0-124-generic-x86_64-with-glibc2.36CPython 3.12.10 †õÒ¨†ƒ‘*-of*>/data/spatop/options_files/sweep_fixed_dp/config_sxwt6rms.json*-l* /scratch/logs*-n*dpfx_blocks_sxwt6rms2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb$dp-spatop-sweepagent-blocks-v2-fb276r/opt/conda/bin/pythonº r92k744tb1j36yx8y01pdz01dvqk7cbqæPò2  requirements.txtŠÊ#ÀW sxwt6rmsZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64l"»œY sxwt6rmsZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64#’„Z* sxwt6rmsb trainer/global_step2uiŽ\  sxwt6rmsZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64‡„†…/  sxwt6rmsb*"trainer/global_step2¼C! J  sxwt6rmsj9 ž†õÒ¸ÛÜ;*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] [y®ïÇ ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€€ý/€à™¡Ç$ª€Àû”ÌÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-ed01d6ad-b673-30ac-3501-763022903663Ê13.2º r92k744tb1j36yx8y01pdz01dvqk7cbq_¡EÀ’  sxwt6rmsj€  ž†õÒ¸‘îYî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ o@Ø[ sxwt6rmsjÉ ž†õÒøôµZ·Trainable params: 5.3 M Non-trainable params: 50 Total params: 5.3 M Total estimated model params size (MB): 21.374 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 þcKp sxwt6rmsj_ ž†õÒðˆÌZPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). εêz sxwt6rmsji ž†õÒ¨ŽÒZZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). qjp sxwt6rmsj_ ž†õÒà™ÖZPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ´O«×s sxwt6rmsjb ž†õÒ ÏÙZSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). !ÁÇßm sxwt6rmsj\ ž†õÒÀåÜZMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). Ü¥&+ sxwt6rms*™  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"\ųªê sxwt6rmsjØ ž†õÒÀ¾ã\È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. 6ÕGÅM sxwt6rmsj» ž†õÒ ïò\«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. z9ùë sxwt6rmsjÙ Ÿ†õÒ¨…Ù©È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. WÎúqP sxwt6rmsj¾ Ÿ†õÒØ­ð©­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. à{X.˜‚(:Ž ­†õÒéè¹, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.71' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.903333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[éP+%‚(:› ­†õÒ§ªŠ-% disk./.usageGB‚1169.7860870361328 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 517.017578125 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚60.55224609375 disk.md0.out‚ 0.46484375( proc.memory.availableMB‚ 327.45703125 disk.dm-1.in‚0 disk.dm-1.out‚ 0.216796875 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 proc.memory.rssMB‚ 2775.34375* proc.memory.percent‚16.939353942871094 network.sent‚33588 network.recv‚38733 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.19515303931821" disk.nvme0n1.out‚ 517.017578125% disk.nvme1c1n1.out‚60.55224609375! disk.dm-0.out‚576.35400390625$ memory_percent‚98.00136089324951 cpu‚26.018951357685722..m3b/ lr-AdamW/pg1"trainer/global_step2:JP€œÝ*3b/ lr-AdamW/pg2"trainer/global_step2:JPÂm¡i-) _wandbruntime‚20  _runtime‚20!"ó”À» _step‚0 & lr-AdamW/pg1‚6.224861390233292e-06 & lr-AdamW/pg2‚6.224861390233292e-06 trainer/global_step‚49 % _timestamp‚1.7844969464380145e+09 _runtime‚ 20.350686532 ’ª5½ & lr-AdamW/pg2‚6.224861390233292e-06 trainer/global_step‚49 % _timestamp‚1.7844969464380145e+09 _runtime‚ 20.350686532 _step‚0 & lr-AdamW/pg1‚6.224861390233292e-06&"×El bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPutµér!bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPX5ep"bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPåŠeOv#br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPpÆs$bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPº=Žžj%bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP(ý±l&bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPvÈBl'bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPq3žk(bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPhn)bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPø™ˆ+g*bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPKFúc+b_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPÚC‚Èi3be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPY]¬f4bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP柘_5b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPK? i6be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP„@×`7b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP«µ!o8bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPF;#k9bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPs;í9a:b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP‡5k;bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP"ó<(nbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPZðp]g?bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP@Ågo@bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPšþàbnAbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPñÎákBbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP7ìçfCbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPnºŒ*iDbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPå£Ë*fEbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPÍ@ÿ™rFbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPcÐqtqGbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP³¿œœ_Hb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP\¤hhIbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP~xÁDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÙ¦k%sLbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP7ÀC‚bMb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPB˜ÑeNba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPÿgÓDeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP;–Çlebh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP~z} nfbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP‹­¹ggbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPÀ_ãÓnhbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPðƒŒ…ribn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP[3âjjbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP´Ø7Inkbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPÕ¢´slbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPIÇ¿}nmbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPEßinbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP—®cfobb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPòe ×hpbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP+hZDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP»Û¢Èh‚bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPò¯r¥tƒbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP äAd„b_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1877736588288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1911930151424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313356800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚370037 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159787520 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1836780868608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1877736588288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 34210602496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1912035004416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313356800 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚340487 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚216684 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚123803 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 34188924504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680879 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚494130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11965 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1852866905712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 34298416128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 34298416128 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11894 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1912035004416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚864167 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ?9DevnÖw ËiceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1852849866352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚370039 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚494130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218108416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179583IÞ¯z'ôN TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚873905 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11894 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1912035004416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚864167 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1852849866352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚370039 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚494130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218108416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179583 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 34102777894 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚340740 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚691 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚216687 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1836998977024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1877736588288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1911930151424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313356800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚370037 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159787520 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1836780868608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1877736588288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 34210602496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1912035004416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313356800 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚340487 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚216684 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚123803 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 34188924504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680879 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚494130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11965 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1852866905712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 34298416128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 34298416128 GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPGŸÖÕl¥bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPgål¦bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPøÉf§ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP*:ƒ-p¨bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP¬Ä‚Ñj©be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPàK¹…oªbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPn1¢f«ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP4Dh¬bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPàsÖo­bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPË´›„j®be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP,Ò£ài¯bd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPcôÀf°ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP‚>æe±b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP¨z_ k²bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPêc3V³bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP1DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP¤ØÖeâb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPd/D`ãb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPÍgäbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP…ZZñiåbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP¢­òŽgæbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP†±Ffgçbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPy) ‡hèbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPu»kqébl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP@X&,mêbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP·fþ—oëbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP2B:¡uìbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP fðb^íbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPš~Z‘`îb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÞœ Ïdïb_ ürõbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPù ß:]öbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPúÂÏÞe÷b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPWV½Þføba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP¥0˜Cnùbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPÏéO×múbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPQ Ëpûbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP¬ŠY*nübi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP¹Á¼.mýbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPžžŠlŠbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPaº. g‹bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPH&¾^jŒbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPNk¨+mbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPEÇÊ;jŽbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPUèÚsbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP]CAœ_bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPR\ƒÁh‘bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPºÝ([f’ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPÅú’m“bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPõq”bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPùî›k•bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPªÖÃfb–b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPoðê+k—bf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPe™†Wc˜b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPù¯Í.™) _wandbruntime‚20  _runtime‚20õC5 ]šúL R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1950079488512 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚891138 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚220974 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚126532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1924603849532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚347509 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚377360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚377358 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1950079488512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚881386 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚126280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1915102955520 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚377358 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1889738973472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 34888712192 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚881386 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚347254 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1949974628352 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚513778 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚49 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 34778729338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_tra˜äM#in_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1915085916160 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚377360 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 36392331776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174323200 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1924500663450 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚220977 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844969466953044e+09 _step‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1873181485056 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 34976532992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚ 20.606542906 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 36218008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0@Ùœ¬„&›þL SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 34976532992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚881386 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚347254 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1949974628352 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚513778 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚49 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 34778729338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1915085916160 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚377360 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 36392331776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174323200 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1924500663450 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚220977 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844969466953044e+09 _step‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1873181485056 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 34976532992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚ 20.606542906 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 36218008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1950079488512 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚891138 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚220974 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚126532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1924603849532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚347509 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚377360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚377358 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1950079488512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚881386 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚126280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1915102955520 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚377358 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1889738973472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 34888712192 E loss/SRqqt1/assignment_loss"trainer/global_step2:JPìý´C¤b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP¶¶ròA¥b< loss/FBt2/assignment_loss"trainer/global_step2:JP‡ì@¦b; loss/FRt2/detection_loss"trainer/global_step2:JPæï—%B§b= loss/SRqqt2/detection_loss"trainer/global_step2:JPØdì7¨b2 loss/total_loss"trainer/global_step2:JPÈ€â|A©b< loss/FRt1/assignment_loss"trainer/global_step2:JPXb1¸.ª) _wandbruntime‚20  _runtime‚20¾m8»ø«ò /loss/FRt2/assignment_loss‚5.287375646584043 /loss/FBt1/assignment_loss‚2.216004166492196 1loss/SRqqt1/detection_loss‚2.5871134930939386 /loss/FRt1/detection_loss‚1.0684245035624231 /loss/FBt1/detection_loss‚15.914632007133129 trainer/global_step‚49 _runtime‚ 20.607594978 .loss/FBt2/detection_loss‚17.69383550151473 epoch‚0 1loss/SRqqt1/assignment_loss‚4.206011833566608 1loss/SRqqt2/assignment_loss‚4.212882201740707 0loss/FBt2/assignment_loss‚2.2070521740686324 .loss/FRt2/detection_loss‚2.543346352177613 _step‚3 1loss/SRqqt2/detection_loss‚2.6953484836587873 %loss/total_loss‚65.38704319008791 % _timestamp‚1.7844969466964731e+09 /loss/FRt1/assignment_loss‚4.755016826495096)"Óü¬ö .loss/FRt2/detection_loss‚2.543346352177613 _step‚3 1loss/SRqqt2/detection_loss‚2.6953484836587873 %loss/total_loss‚65.38704319008791 % _timestamp‚1.7844969466964731e+09 /loss/FRt1/assignment_loss‚4.755016826495096 /loss/FRt2/assignment_loss‚5.287375646584043 /loss/FBt1/assignment_loss‚2.216004166492196 1loss/SRqqt1/detection_loss‚2.5871134930939386 /loss/FRt1/detection_loss‚1.0684245035624231 /loss/FBt1/detection_loss‚15.914632007133129 trainer/global_step‚49 _runtime‚ 20.607594978 .loss/FBt2/detection_loss‚17.69383550151473 epoch‚0 1loss/SRqqt1/assignment_loss‚4.206011833566608 1loss/SRqqt2/assignment_loss‚4.212882201740707 0loss/FBt2/assignment_loss‚2.2070521740686324–zI)­‚(:ž ¼†õÒÔº×+& disk.nvme0c0n1.out‚788.72021484375% disk.nvme1c1n1.out‚100.6298828125 disk.dm-0.in‚0 cpu‚25.973526916155116) disk./.usagePercent‚76.19807062116436% disk./.usageGB‚1169.8308792114258 disk.dm-1.in‚0$ disk.nvme0n1.out‚788.72021484375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚100.6298828125! disk.dm-0.out‚887.45166015625 disk.dm-1.out‚ 0.345703125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 0.72265625$ memory_percent‚98.00596237182617& proc.memory.availableMB‚ 326.703125! proc.memory.rssMB‚ 2775.390625 network.sent‚59519 network.recv‚44598 disk.rbd0.in‚0 disk.nvme0n1.in‚0* proc.memory.percent‚16.939640045166016 proc.cpu.threads‚17S’Ë ›®‚(: ¼†õÒÒ®„, gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚192.833' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.27766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àv/.¯) _wandbruntime‚38  _runtime‚38v¨ŠWð½ trainer/global_step‚99 % _timestamp‚1.7844969650716674e+09 _runtime‚ 38.984615144 _step‚4 ' lr-AdamW/pg1‚1.2576760768022365e-05 ' lr-AdamW/pg2‚1.2576760768022365e-05¦Tn°Ç±Á ' lr-AdamW/pg2‚1.2576760768022365e-05 trainer/global_step‚99 % _timestamp‚1.7844969650716674e+09 _runtime‚ 38.984615144 _step‚4 ' lr-AdamW/pg1‚1.2576760768022365e-05–hJ.²) _wandbruntime‚38  _runtime‚38_îæ³|'³öN F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 68236157952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3746037910528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1000618 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚679113 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176427520 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 70488423424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1000618 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3586026181632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3814186254848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193843712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3656715936256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3814186254848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚736089 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚248184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 70664850944 _step‚5 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚736087 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7844969650743382e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3746054949888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚990882 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3656514605056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 68148344320 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚990882 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚679371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201331200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0‚&êsâ´úN WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3764488693624 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 68018399912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3814291107840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚736087 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚990882 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3656514605056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 68148344320 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚990882 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚679371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201331200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 68236157952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3746037910528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1000618 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚679113 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176427520 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 70488423424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1000618 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3586026181632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3814186254848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193843712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3656715936256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3814186254848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚736089 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚248184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 70664850944 _step‚5 J?DeviceStatsMonitor.on_train_batch_start/active.largKÄ”{ž e_pool.freed‚736087 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7844969650743382e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3746054949888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3852335591936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 68826454016 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1744188 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1000780 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1000780 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚435474 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚250654 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3692904406528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844969653285491e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚699 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚685875 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1753940 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1010530 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 68694351468 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 39.239812635 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3783421317120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Oç}Œ&·†M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 68608204746 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1010530 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 68694351468 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 39.239812635 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3783421317120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1010530 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3801933526858 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚250401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3621721338880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3733325322112 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184808960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚743410 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚743408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3852335591936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 68826454016 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1744188 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1000780 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1000780 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚435474 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚250654 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3692904406528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844969653285491e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚699 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚685875 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1753940 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5641921557032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5614356272128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5540056642352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5614373311488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1102137 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 102173899776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚149 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1102139 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5614373311488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚371662 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1102137 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5716442358272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5716547211264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 L@DeviceStatsMonitor.on_train¿›‚QŠ_batch_start/inactive_split.all.freed‚1017346 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5476410561536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 101847875320 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102173899776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚645687 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 105601613824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5614356272128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209719808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5641921557032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5614356272128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5540056642352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5614373311488 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1102137 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 102173899776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚149 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1102139 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5614373311488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚371662 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1102137 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5716442358272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5716547211264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1017346 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5476410561536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 101847875320 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102173899776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚645687 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 105601613824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5614356272128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209719808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 102852016640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1109458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844969839782696e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚649974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102764195840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5406479201280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5576945749472 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2616742 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5512597430272 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚149 _step‚10 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1109458 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1497532 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚649977 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102437680154 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5754591695360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5651739678720 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1507282 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102764195840 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1024093 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5754486835200 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 106118228992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1109460 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5651739678720 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 102852016640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 102523826876 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1109460 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚699 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5512805045760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5651722639360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚374119 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ˜ÕFü!Å–M MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1507282 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102764195840 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1024093 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5754486835200 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 106118228992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1109460 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5651739678720 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 102852016640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 102523826876 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1109460 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚699 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5512805045760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5651722639360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚374119 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 102852016640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1109458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844969839782696e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚649974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102764195840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5406479201280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5576945749472 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2616742 C9DeviceStatsMonitor.on_train_batch_end/allocatWHò ion.all.peak‚11967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5512597430272 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚149 _step‚10 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1109458 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1497532 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚649977 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102437680154 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5754591695360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5651739678720 ãÑ‚©.Æ) _wandbruntime‚57  _runtime‚57…‰¯ùÇó .loss/FBt2/detection_loss‚12.20030608975835 _step‚11 1loss/SRqqt2/assignment_loss‚4.174567205182622 .loss/FRt1/detection_loss‚0.554998790748675 _runtime‚ 57.890053925 1loss/SRqqt1/assignment_loss‚4.204564723415651 1loss/SRqqt2/detection_loss‚1.6778323999449107 %loss/total_loss‚70.85525484910909 epoch‚0 trainer/global_step‚149 /loss/FRt1/assignment_loss‚4.541507393755811 /loss/FRt2/assignment_loss‚5.296888153274338 0loss/FBt1/assignment_loss‚2.0772192726532617 /loss/FRt2/detection_loss‚1.5691304422818222 0loss/SRqqt1/detection_loss‚5.038118785490161 /loss/FBt1/detection_loss‚27.415268023187917 $ _timestamp‚1.784496983979328e+09 0loss/FBt2/assignment_loss‚2.1048535694155777âç¥{ýÈ÷ 1loss/SRqqt1/assignment_loss‚4.204564723415651 1loss/SRqqt2/detection_loss‚1.6778323999449107 %loss/total_loss‚70.85525484910909 epoch‚0 trainer/global_step‚149 /loss/FRt1/assignment_loss‚4.541507393755811 /loss/FRt2/assignment_loss‚5.296888153274338 0loss/FBt1/assignment_loss‚2.0772192726532617 /loss/FRt2/detection_loss‚1.5691304422818222 0loss/SRqqt1/detection_loss‚5.038118785490161 /loss/FBt1/detection_loss‚27.415268023187917 $ _timestamp‚1.784496983979328e+09 0loss/FBt2/assignment_loss‚2.1048535694155777 .loss/FBt2/detection_loss‚12.20030608975835 _step‚11 1loss/SRqqt2/assignment_loss‚4.174567205182622 .loss/FRt1/detection_loss‚0.554998790748675 _runtime‚ 57.890053925 xê—-É‚(:¢ Ú†õÒ“¼ì+' proc.memory.availableMB‚ 326.1171875! proc.memory.rssMB‚ 2775.390625 network.recv‚53586) disk./.usagePercent‚76.19807385132651% disk./.usageGB‚1169.8309288024902 disk.md0.in‚0! disk.dm-0.out‚891.24365234375& disk.nvme0c0n1.out‚792.98681640625# memory_percent‚98.0095386505127* proc.memory.percent‚16.939640045166016 disk.dm-1.in‚0 disk.dm-1.out‚ 0.75341796875 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚102.03564453125 cpu‚25.92955195635113 proc.cpu.threads‚17 disk.md0.out‚ 1.39453125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚102.03564453125 disk.nvme1n1.in‚0 network.sent‚140955 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚792.98681640625Ú æ3™Ê‚(:Ž Ú†õÒ̈- gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.641' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.21366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÄV+Ë‚(:  é†õÒÝ›»+ cpu‚25.92191940846026 network.sent‚189732) disk./.usagePercent‚76.19812826713509% disk./.usageGB‚1169.8317642211914 disk.nvme0n1.in‚0 disk.md0.out‚ 1.78515625& disk.nvme0c0n1.out‚798.30029296875* proc.memory.percent‚16.939640045166016 network.recv‚58559 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 0.994140625# memory_percent‚98.0149507522583( proc.memory.availableMB‚ 325.23046875! proc.memory.rssMB‚ 2775.390625$ disk.nvme0n1.out‚798.30029296875& disk.nvme1c1n1.out‚103.43017578125$ disk.nvme1n1.out‚103.43017578125 disk.dm-0.out‚896.8798828125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0"Á¢ì›Ì‚(: é†õÒ¡³ã+ gpu.0.gpu‚59.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚153.352' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚51.117333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕxT.Í) _wandbruntime‚76  _runtime‚76àÓ­ÀÅο % _timestamp‚1.7844970024128942e+09 _runtime‚ 76.325558395 _step‚12 ' lr-AdamW/pg1‚2.5280559523600512e-05 ' lr-AdamW/pg2‚2.5280559523600512e-05 trainer/global_step‚199ÉKw1ÉÏà _step‚12 ' lr-AdamW/pg1‚2.5280559523600512e-05 ' lr-AdamW/pg2‚2.5280559523600512e-05 trainer/global_step‚199 % _timestamp‚1.7844970024128942e+09 _runtime‚ 76.325558395 ƒm1.Ð) _wandbruntime‚76  _runtime‚76¨öÇVŽ'шO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7618803314688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 140343866368 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7618698461696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7519354420440 _step‚13 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7482674633728 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 136111641600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7296089201152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1994122 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1984386 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3452573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7383660030352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7482674633728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚496381 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 % _timestamp‚1.7844970024152646e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1356565 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1468187 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚708 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178524672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1468189 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7383677069712 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 135677350728 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 136023827968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 136023827968 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3462311 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 135591204118 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚199 _runtime‚ 76.326105726 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7482691673088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1468187 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7618698461696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7295885772800 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7519251234470 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1984386 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚860187 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1356835 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚496648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7618803314688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203428352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104b~9û’'ÒŒO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1468187 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7618698461696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7295885772800 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7519251234470 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1984386 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚860187 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1356835 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚496648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7618803314688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203428352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7618803314688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 140343866368 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7618698461696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7519354420440 _step‚13 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7482674633728 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 136111641600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7296089201152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1994122 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1984386 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3452573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7383660030352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7482674633728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚496381 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 % _timestamp‚1.7844970024152646e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1356565 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1468187 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚708 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178524672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1468189 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7383677069712 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 135677350728 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 136023827968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 136023827968 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3462311 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 135591204118 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚199 _runtime‚ 76.326105726 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7482691673088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 Ä.Ó) _wandbruntime‚76  _runtime‚76觢œÔ’M UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 136789758464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7191261967360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚14 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7656742938624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7656847798784 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚498894 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7332483222528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 136353302284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7656847798784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7420549137472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1363368 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚499161 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 141036446208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚864477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844970026701858e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1475508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7556799253674 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 136789758464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3479544 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3469792 =7DeviceStatsMonitoräm .on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3479544 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3469792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 136267155562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚14 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7656742938624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7656847798784 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚498894 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7332483222528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 136353302284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7656847798784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7420549137472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1363368 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚499161 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 141036446208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚864477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844970026701858e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1475508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7556799253674 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 136789758464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3479544 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3469792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3479544 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3469792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 136267155562 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 136789758464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482©™N.Ö) _wandbruntime‚76  _runtime‚76œ†©gû×õ 1loss/SRqqt2/detection_loss‚1.3763804444384695 _step‚15 0loss/FBt1/assignment_loss‚1.7842308044433595 0loss/FRt1/detection_loss‚0.48961109907161127 1loss/SRqqt1/detection_loss‚5.2653411816466935 .loss/FBt2/detection_loss‚9.891020835721148 epoch‚0 trainer/global_step‚199 1loss/SRqqt1/assignment_loss‚4.276357939749053 1loss/SRqqt2/assignment_loss‚4.215190185440911 0loss/FBt2/assignment_loss‚1.8623334279427162 %loss/total_loss‚84.11333022711128 $ _timestamp‚1.784497002671201e+09 _runtime‚ 76.581970385 /loss/FRt2/detection_loss‚1.4798964168534192 .loss/FBt1/detection_loss‚43.63255971471469 /loss/FRt1/assignment_loss‚4.537214820667849 /loss/FRt2/assignment_loss‚5.303193356421363„Ã::ÿØù 1loss/SRqqt1/assignment_loss‚4.276357939749053 1loss/SRqqt2/assignment_loss‚4.215190185440911 0loss/FBt2/assignment_loss‚1.8623334279427162 %loss/total_loss‚84.11333022711128 $ _timestamp‚1.784497002671201e+09 _runtime‚ 76.581970385 /loss/FRt2/detection_loss‚1.4798964168534192 .loss/FBt1/detection_loss‚43.63255971471469 /loss/FRt1/assignment_loss‚4.537214820667849 /loss/FRt2/assignment_loss‚5.303193356421363 1loss/SRqqt2/detection_loss‚1.3763804444384695 _step‚15 0loss/FBt1/assignment_loss‚1.7842308044433595 0loss/FRt1/detection_loss‚0.48961109907161127 1loss/SRqqt1/detection_loss‚5.2653411816466935 .loss/FBt2/detection_loss‚9.891020835721148 epoch‚0 trainer/global_step‚199G O",Ù‚(:¡ ø†õÒÕ¸Å+! proc.memory.rssMB‚ 2775.390625% disk./.usageGB‚1165.7797813415527 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.01597595214844 cpu‚25.921113694545404) disk./.usagePercent‚75.93419842641606$ disk.nvme0n1.out‚805.50927734375! disk.dm-0.out‚905.51708984375 disk.dm-1.out‚ 1.19482421875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚105.95263671875 disk.md0.in‚0 disk.md0.out‚ 2.20703125 disk.rbd0.in‚0* proc.memory.percent‚16.939640045166016 proc.cpu.threads‚17 network.sent‚193224 network.recv‚62452 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚105.95263671875 disk.dm-0.in‚0& disk.nvme0c0n1.out‚805.50927734375$ proc.memory.availableMB‚325.0625*ßh›Ú‚(: ø†õÒñ²“, gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.657' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.55233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qõSk.Û) _wandbruntime‚95  _runtime‚950üššÃܽ % _timestamp‚1.7844970210921128e+09 _runtime‚ 95.004585065 _step‚16 & lr-AdamW/pg1‚3.163245890138958e-05 & lr-AdamW/pg2‚3.163245890138958e-05 trainer/global_step‚249R`×ÇÝÁ % _timestamp‚1.7844970210921128e+09 _runtime‚ 95.004585065 _step‚16 & lr-AdamW/pg1‚3.163245890138958e-05 & lr-AdamW/pg2‚3.163245890138958e-05 trainer/global_step‚249Îv+.Þ) _wandbruntime‚95  _runtime‚95H èP'ߊO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2481138 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2481138 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220205568 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175450241536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9227280457712 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4315375 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 169420679526 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 170049383424 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4325113 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2490874 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1696986 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8940324672512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9227263418352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚622022 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9521059418112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1074687 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 % _timestamp‚1.7844970210943103e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚622299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 169506826136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175254939648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1834239 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 169961569792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1696706 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9115774914048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9520954565120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9351010034688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _runtime‚ 95.005173516 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2490874 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚17 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 169961569792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2481138 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2481138 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220205568 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175450241536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9227280457712 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4315375 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 169420679526 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 170049383424 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4325113 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2490874 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1696986 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8940324672512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9227263418352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚622022 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9521059418112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1074687 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 % _timestamp‚1.7844970210943103e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚622299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚27‹ía¾€7 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 169506826136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175254939648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1834239 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 169961569792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1696706 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9115774914048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9520954565120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9351010034688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _runtime‚ 95.005173516 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2490874 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚17 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 169961569792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 175954863616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4342346 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2491036 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9151949789696 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9434335303164 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2500786 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1703499 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9264135486112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 170182777692 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184808960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8976019829760 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 170096630970 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚ 95.259208686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9152159502336 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9559103902208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1078974 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚28Ò1ž&ã˜M WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 170096630970 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚ 95.259208686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9152159502336 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9559103902208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1078974 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1841560 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4332594 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1841558 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 170639679488 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2500786 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 175954863616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4342346 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2491036 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9151949789696 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9434335303164 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2500786 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1703499 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9264135486112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 170182777692 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184808960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8976019829760b”àá.ä) _wandbruntime‚95  _runtime‚95¸OÝøåò /loss/FRt1/assignment_loss‚4.545420782717971 trainer/global_step‚249 $ _timestamp‚1.784497021349105e+09 _runtime‚ 95.259715527 _step‚19 .loss/FBt2/detection_loss‚6.637961059189653 epoch‚0 /loss/FRt2/assignment_loss‚5.361615216860208 /loss/SRqqt2/assignment_loss‚4.1608792998574 0loss/FBt1/assignment_loss‚1.2727935446633234 0loss/FBt2/assignment_loss‚1.2555782764772825 0loss/SRqqt1/detection_loss‚5.681409763526325 /loss/FRt1/detection_loss‚0.4327282115786112 /loss/FRt2/detection_loss‚1.4722442842058596 &loss/total_loss‚108.22734530263594 1loss/SRqqt1/assignment_loss‚4.219423299978588 1loss/SRqqt2/detection_loss‚1.3610292716459795 .loss/FBt1/detection_loss‚71.82626229193475ö48ëüæö 0loss/FBt2/assignment_loss‚1.2555782764772825 0loss/SRqqt1/detection_loss‚5.681409763526325 /loss/FRt1/detection_loss‚0.4327282115786112 /loss/FRt2/detection_loss‚1.4722442842058596 &loss/total_loss‚108.22734530263594 1loss/SRqqt1/assignment_loss‚4.219423299978588 1loss/SRqqt2/detection_loss‚1.3610292716459795 .loss/FBt1/detection_loss‚71.82626229193475 /loss/FRt1/assignment_loss‚4.545420782717971 trainer/global_step‚249 $ _timestamp‚1.784497021349105e+09 _runtime‚ 95.259715527 _step‚19 .loss/FBt2/detection_loss‚6.637961059189653 epoch‚0 /loss/FRt2/assignment_loss‚5.361615216860208 /loss/SRqqt2/assignment_loss‚4.1608792998574 0loss/FBt1/assignment_loss‚1.2727935446633234 ÀG'ç‚(:œ ‡‡õÒûû¾+% disk./.usageGB‚1165.7799263000488 disk.rbd0.in‚0# memory_percent‚98.0036735534668& proc.memory.availableMB‚ 327.078125 cpu‚25.916338543332223 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚821.47021484375$ disk.nvme0n1.out‚821.47021484375 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 110.162109375 disk.nvme1n1.in‚0 network.recv‚67141 disk.dm-1.out‚ 1.810546875 disk.md0.out‚ 4.94140625! proc.memory.rssMB‚ 2775.390625* proc.memory.percent‚16.939640045166016 proc.cpu.threads‚17 network.sent‚241193! disk.dm-0.out‚919.51708984375 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0" disk.nvme1n1.out‚ 110.162109375 disk.md0.in‚0) disk./.usagePercent‚75.93420786842852_oè‚(:… ‡‡õÒû³, gpu.0.gpu‚77.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚146.487' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚48.829 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öX$¨0é+ _wandbruntime‚113  _runtime‚113KÜÅÇÆêÀ trainer/global_step‚299 % _timestamp‚1.7844970397779112e+09 _runtime‚ 113.690605155 _step‚20 ' lr-AdamW/pg1‚3.7984358279178665e-05 ' lr-AdamW/pg2‚3.7984358279178665e-05ßÉ+ŸÊëÄ ' lr-AdamW/pg2‚3.7984358279178665e-05 trainer/global_step‚299 % _timestamp‚1.7844970397779112e+09 _runtime‚ 113.690605155 _step‚20 ' lr-AdamW/pg1‚3.7984358279178665e-05FxÈ¿0ì+ _wandbruntime‚113  _runtime‚113S^qïíšO YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203250154934 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5187915 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2200289 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10935446897664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2987626 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11423315521536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 203987125248 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2200287 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2987626 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11274220147256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚747013 GADeviceStatsMonitor.on_train_batch_start/overs•žB¾±ize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11070883845712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11219311356928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11423210668544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10725082534912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5178177 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 203899311616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11219311356928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2977890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2200287 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 113.691157586 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11423210668544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10725057631232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10935239275008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚21 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚747013 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11070883845712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11219311356928 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11423210668544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10725082534912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5178177 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 203899311616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11219311356928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2977890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚299 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2200287 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 113.691157586 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11423210668544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10725057631232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10935239275008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚21 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11311768166572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2042943 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚299 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11461360005632 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5205148 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5195396 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚749728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11107755913472 _runtime‚ 113.946156342 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204665242112 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2997538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204012253100 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2207610 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 210876920832 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 211065924096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚749469 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11256677724160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10971843616256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10971629709312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204577421312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11461360005632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2987788 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11256694763520 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1293474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2207610 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10760777692160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚22 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2207608 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2997538 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11256677724160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204577421312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11461255145472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11256694763520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189003264 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2207608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213906944 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 203926106378 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204665242112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11107738874112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482=îR ­ñ©M =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2207610 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10760777692160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚22 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2207608 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2997538 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11256677724160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204577421312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11461255145472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11256694763520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189003264 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2207608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213906944 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 203926106378 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204665242112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11107738874112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5195396 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2987788 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1293477 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11311664980490 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11311768166572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2042943 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚299 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11461360005632 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5205148 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5195396 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚749728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11107755913472 _runtime‚ 113.946156342 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204665242112 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2997538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204012253100 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2207610 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 210876920832 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 211065924096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚749469 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11256677724160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10971843616256 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10971629709312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 súk6GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204577421312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11461360005632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2987788 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11256694763520 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1293474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6040979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 _step‚25 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3484378 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚872592 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 245116833792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12914487233712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2566337 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13325466771968 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6050717 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13325571624960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12509815493632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13151653010664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 237924867072 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12509840397312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13325571624960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13325466771968 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2376279 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 245301649920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6040979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 _step‚25 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3484378 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚872592 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 245116833792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12914487233712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2566337 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13325466771968 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6050717 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13325571624960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12509815493632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13151653010664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 237924867072 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12509840397312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13325571624960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13325466771968 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2376279 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 245301649920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚874810 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245809393152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 132.765479178 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 238602983936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12545510650880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 237755581786 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2573658 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚349 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2573660 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 238602983936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245998396416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12951342262112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 238515163136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6067950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13124996085760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3484540 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1507974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189003264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2382784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6058198 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3484540 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12791320044032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844970588543875e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13124996085760 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12791533950976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13189201029980 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2573658 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13125013125120 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2383041 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 H@DeviceStat ªæX sMonitor.on_train_batch_end/inactive_split.all.current‚257 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13363616109056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6067950 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12545535554560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚875064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 237841728508 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3494290 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 238602983936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245998396416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12951342262112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 238515163136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6067950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13124996085760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3484540 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1507974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189003264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2382784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6058198 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3484540 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12791320044032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844970588543875e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13124996085760 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12791533950976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13189201029980 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2573658 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13125013125120 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2383041 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13363616109056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6067950 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12545535554560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚875064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 237841728508 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3494290 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚874810 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245809393152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 132.765479178 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 238602983936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12545510650880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 237755581786Í~Ý0€+ _wandbruntime‚132  _runtime‚132°…¯üö trainer/global_step‚349 _runtime‚ 132.765940279 1loss/SRqqt2/detection_loss‚1.1765224121447595 .loss/FBt2/detection_loss‚4.889397407480112 /loss/FRt1/assignment_loss‚4.507636628943503 /loss/FRt2/assignment_loss‚5.370031901768276 2loss/SRqqt1/assignment_loss‚2.8497789319484466 1loss/SRqqt2/assignment_loss‚2.688720504650895 0loss/SRqqt1/detection_loss‚5.339853336613986 %loss/total_loss‚64.39691714157806 epoch‚0 % _timestamp‚1.7844970588553004e+09 0loss/FBt1/assignment_loss‚0.5291231015904082 1loss/FBt2/assignment_loss‚0.26773481216663736 .loss/FRt1/detection_loss‚0.380472280498726 /loss/FRt2/detection_loss‚1.5008802576517524 .loss/FBt1/detection_loss‚34.89676556612054 _step‚27ÝŽž%‚ú _step‚27 trainer/global_step‚349 _runtime‚ 132.765940279 1loss/SRqqt2/detection_loss‚1.1765224121447595 .loss/FBt2/detection_loss‚4.889397407480112 /loss/FRt1/assignment_loss‚4.507636628943503 /loss/FRt2/assignment_loss‚5.370031901768276 2loss/SRqqt1/assignment_loss‚2.8497789319484466 1loss/SRqqt2/assignment_loss‚2.688720504650895 0loss/SRqqt1/detection_loss‚5.339853336613986 %loss/total_loss‚64.39691714157806 epoch‚0 % _timestamp‚1.7844970588553004e+09 0loss/FBt1/assignment_loss‚0.5291231015904082 1loss/FBt2/assignment_loss‚0.26773481216663736 .loss/FRt1/detection_loss‚0.380472280498726 /loss/FRt2/detection_loss‚1.5008802576517524 .loss/FBt1/detection_loss‚34.89676556612054Áh ï)ƒ‚(:ž ¥‡õÒÏøæ+ disk.nvme0n1.in‚0! disk.dm-0.out‚929.35888671875 disk.dm-1.out‚ 2.1787109375 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 831.900390625! proc.memory.rssMB‚ 2775.390625 cpu‚25.924729869226752 disk.md0.out‚ 5.8203125 network.recv‚76748) disk./.usagePercent‚75.93420612911044$ disk.nvme1n1.out‚111.81396484375 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0$ memory_percent‚98.00598621368408( proc.memory.availableMB‚ 326.69921875 disk.md0.in‚0 disk.dm-0.in‚0$ disk./.usageGB‚1165.779899597168" disk.nvme0n1.out‚ 831.900390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚111.81396484375 disk.nvme1n1.in‚0* proc.memory.percent‚16.939640045166016 network.sent‚347981T‰q™„‚(:Ž ¥‡õÒ¤–, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.19566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÒ´¸-…‚(:¢ ´‡õÒÍÄ·+* proc.memory.percent‚16.939640045166016 cpu‚25.924111930367815 disk.md0.out‚ 6.43359375 network.recv‚81312 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚112.6474609375 disk.dm-1.in‚0& disk.nvme0c0n1.out‚833.11279296875( proc.memory.availableMB‚ 326.62890625 network.sent‚382882) disk./.usagePercent‚75.93421010469463 disk.md0.in‚0! disk.dm-0.out‚930.04638671875 disk.dm-1.out‚ 2.2373046875 disk.rbd0.in‚0! proc.memory.rssMB‚ 2775.390625 disk.dm-0.in‚0% disk./.usageGB‚1165.7799606323242$ disk.nvme0n1.out‚833.11279296875 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚112.6474609375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17$ memory_percent‚98.00641536712646™“pT›†‚(: ´‡õÒ€Ïß+ gpu.0.gpu‚87.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚194.555' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.85166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Fð0‡+ _wandbruntime‚151  _runtime‚151g…PĈ¾ & lr-AdamW/pg1‚5.068815703475681e-05 & lr-AdamW/pg2‚5.068815703475681e-05 trainer/global_step‚399 % _timestamp‚1.7844970773276663e+09 _runtime‚ 151.238606338 _step‚28ûzûȉ _runtime‚ 151.238606338 _step‚28 & lr-AdamW/pg1‚5.068815703475681e-05 & lr-AdamW/pg2‚5.068815703475681e-05 trainer/global_step‚399 % _timestamp‚1.7844970773276663e+094a€0Š+ _wandbruntime‚151  _runtime‚151ÏÊì¡'‹›O ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14574806192640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218108416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 280207932928 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280014728192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2932387 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6913519 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3981130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1718187 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14294598259712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15028982688102 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2715532 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚997093 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6903781 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 271774795264 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844970773285458e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3981130 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3971394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3971394 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14758073582352 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15227722875392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 271862608896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14955948080128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚399 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 271862608896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 193204736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15227827728384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14955948080128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14758090621712 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 270995252360 _runtime‚ 151.239367259 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14574588084224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14574588084224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BâX 218108416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 280207932928 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280014728192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2932387 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6913519 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3981130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1718187 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14294598259712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15028982688102 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2715532 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚997093 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6903781 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 271774795264 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844970773285458e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3981130 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3971394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3971394 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14758073582352 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15227722875392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0NWžß0+ _wandbruntime‚151  _runtime‚151¦•«&Ž¥M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 % _timestamp‚1.7844970775864975e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 272452904960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15066633893388 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6921000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 272540725760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15066530707306 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14993331486720 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6921000 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 271585057194 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14993314447360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6930752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2939708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6930752 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3991042 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1722474 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2939708 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3981292 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2939710 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 280899452928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15265872212480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 272540725760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14330293416960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 272452904960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚999811 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1722477 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2722030 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14794945650112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚399 _step‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14330268513280 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [¯¯&©M LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2939710 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 280899452928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15265872212480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 272540725760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14330293416960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 272452904960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚999811 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1722477 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2722030 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14794945650112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚399 _step‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14330268513280 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 % _timestamp‚1.7844970775864975e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 272452904960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15066633893388 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6921000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 272540725760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15066530707306 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14993331486720 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6921000 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 271585057194 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14993314447360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6930752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2939708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6930752 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3991042 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1722474 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2939708 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3981292A‰}0+ _wandbruntime‚151  _runtime‚151I û÷‘ñ $ _timestamp‚1.784497077587408e+09 1loss/SRqqt1/assignment_loss‚2.612848556402958 0loss/SRqqt1/detection_loss‚4.802470481248967 .loss/FBt1/detection_loss‚34.29753518187337 _runtime‚ 151.4987227 _step‚31 2loss/SRqqt2/assignment_loss‚2.4689214087050892 0loss/FBt1/assignment_loss‚0.6199480897436539 /loss/FRt2/detection_loss‚1.4694947969814962 0loss/SRqqt2/detection_loss‚1.173428299137057 epoch‚0 trainer/global_step‚399 %loss/total_loss‚63.87976697751783 .loss/FRt1/assignment_loss‚4.55328968192349 /loss/FRt2/assignment_loss‚5.314748748274874 0loss/FBt2/assignment_loss‚0.1381052012865742 /loss/FRt1/detection_loss‚0.3965026482206662 .loss/FBt2/detection_loss‚6.032473883719632¬3°¹û’õ 1loss/SRqqt1/assignment_loss‚2.612848556402958 0loss/SRqqt1/detection_loss‚4.802470481248967 .loss/FBt1/detection_loss‚34.29753518187337 _runtime‚ 151.4987227 _step‚31 2loss/SRqqt2/assignment_loss‚2.4689214087050892 0loss/FBt1/assignment_loss‚0.6199480897436539 /loss/FRt2/detection_loss‚1.4694947969814962 0loss/SRqqt2/detection_loss‚1.173428299137057 epoch‚0 trainer/global_step‚399 %loss/total_loss‚63.87976697751783 .loss/FRt1/assignment_loss‚4.55328968192349 /loss/FRt2/assignment_loss‚5.314748748274874 0loss/FBt2/assignment_loss‚0.1381052012865742 /loss/FRt1/detection_loss‚0.3965026482206662 .loss/FBt2/detection_loss‚6.032473883719632 $ _timestamp‚1.784497077587408e+09 uüÎ+“‚(:  ÇõÒŒá²+% disk./.usageGB‚1165.7799491882324 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.md0.out‚ 6.8359375$ memory_percent‚98.01130294799805 network.sent‚386475 disk.rbd0.out‚0 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚113.72705078125 disk.dm-0.in‚0! disk.dm-0.out‚932.41748046875) disk./.usagePercent‚75.93420935927259 disk.rbd0.in‚0$ disk.nvme0n1.out‚835.58642578125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚113.72705078125& proc.memory.availableMB‚ 325.828125* proc.memory.percent‚16.939640045166016 proc.cpu.threads‚17 network.recv‚85623 disk.dm-1.out‚ 2.55029296875& disk.nvme0c0n1.out‚835.58642578125 disk.nvme1c1n1.in‚0 disk.md0.in‚0! proc.memory.rssMB‚ 2775.390625 cpu‚25.9234965417118xJšB›”‚(: ÇõÒ¹¥„, gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚192.832' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.27733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O"o€0•+ _wandbruntime‚169  _runtime‚169€>’Ö½ & lr-AdamW/pg1‚5.704005641254588e-05 & lr-AdamW/pg2‚5.704005641254588e-05 trainer/global_step‚449 $ _timestamp‚1.784497096068925e+09 _runtime‚ 169.979872192 _step‚32I…ÔtÇ—Á trainer/global_step‚449 $ _timestamp‚1.784497096068925e+09 _runtime‚ 169.979872192 _step‚32 & lr-AdamW/pg1‚5.704005641254588e-05 & lr-AdamW/pg2‚5.704005641254588e-05 UU©j0˜+ _wandbruntime‚169  _runtime‚1696Û¿Rê™O @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16824283481088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17129978978816 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 305712537088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17129978978816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16394509435904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3055003 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚33 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7766583 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 304738581158 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚449 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3298437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.pÛúªþ¹eak‚ 6159788544 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305800350720 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4477882 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16601694009712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16601676970352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16079356122112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3298439 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1932684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16079331218432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7776321 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3054735 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 169.982958917 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4468146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305800350720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 304824727768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16824266441728 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 305712537088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17130083831808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1932687 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4468146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1122051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16906518737480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 314970594816ã &®§'š¡O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16906518737480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 314970594816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16824283481088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17129978978816 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 305712537088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17129978978816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16394509435904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3055003 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚33 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7766583 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 304738581158 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚449 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3298437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305800350720 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4477882 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16601694009712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16601676970352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16079356122112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3298439 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1932684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16079331218432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7776321 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3054735 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 169.982958917 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4468146 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305800350720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 304824727768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16824266441728 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 305712537088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17130083831808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1932687 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4468146 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1122051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970!„gÁ 0›+ _wandbruntime‚170  _runtime‚170n p ­&œ§M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4487794 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17168128315904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16115026375680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4478044 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 315673955328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16861632808960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16944066756796 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3305758 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16861649848320 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1124826 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305500679324 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17168128315904 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3061803 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1124567 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7793554 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16861649848320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 315850375680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7793554 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3305760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3305758 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16943963570714 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3305760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1936974 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4478044 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 306478467584 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17168023455744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16861632808960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 trainer/global_step‚449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 306478467584 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3061541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201324032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4487794 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16430700331008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16638566077472 _runtime‚ 170.238931412 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176420352 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305414532602 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7783802 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 _step‚34 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330¨è 0 «M R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16943963570714 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3305760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1936974 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4478044 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 306478467584 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17168023455744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16861632808960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 trainer/global_step‚449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 306478467584 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3061541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201324032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4487794 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16430700331008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16638566077472 _runtime‚ 170.238931412 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176420352 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305414532602 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7783802 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 _step‚34 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4487794 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17168128315904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16115026375680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4478044 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 315673955328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16861632808960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.p‰åÙy eak‚1362 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16944066756796 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3305758 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16861649848320 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1124826 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305500679324 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17168128315904 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3061803 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1124567 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7793554 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16861649848320 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 315850375680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7793554 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3305760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3305758"4G-0ž+ _wandbruntime‚170  _runtime‚170ÍÆ€‡ûŸõ /loss/FBt1/detection_loss‚21.841033238385403 epoch‚0 trainer/global_step‚449 0loss/FRt1/assignment_loss‚4.5609381392944695 1loss/SRqqt1/assignment_loss‚2.542073876890418 $ _timestamp‚1.784497096328767e+09 _step‚35 /loss/FBt1/assignment_loss‚0.313651091651991 0loss/FBt2/assignment_loss‚0.2685174388170708 .loss/FBt2/detection_loss‚6.301418483513054 0loss/FRt2/assignment_loss‚4.9146363382215625 2loss/SRqqt2/assignment_loss‚2.2306858431209218 /loss/FRt1/detection_loss‚0.4873192270032907 %loss/total_loss‚50.26567197609337 _runtime‚ 170.239433723 -loss/FRt2/detection_loss‚1.38733795859984 0loss/SRqqt1/detection_loss‚4.370155001390878 1loss/SRqqt2/detection_loss‚1.04790533920447644ßÚ"ÿ ù -loss/FRt2/detection_loss‚1.38733795859984 0loss/SRqqt1/detection_loss‚4.370155001390878 1loss/SRqqt2/detection_loss‚1.0479053392044764 /loss/FBt1/detection_loss‚21.841033238385403 epoch‚0 trainer/global_step‚449 0loss/FRt1/assignment_loss‚4.5609381392944695 1loss/SRqqt1/assignment_loss‚2.542073876890418 $ _timestamp‚1.784497096328767e+09 _step‚35 /loss/FBt1/assignment_loss‚0.313651091651991 0loss/FBt2/assignment_loss‚0.2685174388170708 .loss/FBt2/detection_loss‚6.301418483513054 0loss/FRt2/assignment_loss‚4.9146363382215625 2loss/SRqqt2/assignment_loss‚2.2306858431209218 /loss/FRt1/detection_loss‚0.4873192270032907 %loss/total_loss‚50.26567197609337 _runtime‚ 170.239433723#6 -¡‚(:¢ Ò‡õÒ¨«Î+# disk.nvme0n1.out‚837.0810546875* proc.memory.percent‚16.939640045166016 proc.cpu.threads‚17% disk./.usageGB‚1165.7802810668945 disk.md0.out‚ 7.23828125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚837.0810546875 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚114.64208984375 disk.nvme1n1.in‚0 network.recv‚90222) disk./.usagePercent‚75.93423097651161 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ disk.nvme1n1.out‚114.64208984375( proc.memory.availableMB‚ 325.50390625! proc.memory.rssMB‚ 2775.390625$ memory_percent‚98.01328182220459 cpu‚25.922905179189964 network.sent‚447650 disk.md0.in‚0 disk.dm-0.out‚933.8251953125 disk.dm-1.out‚ 2.68310546875 disk.rbd0.in‚0ªG¾›¢‚(: Ò‡õÒ ×õ+ gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚196.265' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.42166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H2øh0£+ _wandbruntime‚188  _runtime‚188m¢¬jĤ¾ & lr-AdamW/pg1‚6.339195579033494e-05 & lr-AdamW/pg2‚6.339195579033494e-05 trainer/global_step‚499 % _timestamp‚1.7844971148285317e+09 _runtime‚ 188.741411202 _step‚36Tn¾ãȥ & lr-AdamW/pg1‚6.339195579033494e-05 & lr-AdamW/pg2‚6.339195579033494e-05 trainer/global_step‚499 % _timestamp‚1.7844971148285317e+09 _runtime‚ 188.741411202 _step‚36$ø0¦+ _wandbruntime‚188  _runtime‚188/ìÞù¢'§œO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189010432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚37 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19032235082240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18692584803328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 339650278912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19032339935232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3394006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18783951600888 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4974634 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17864089080832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 188.741476222 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18214000907264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3664489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19032235082240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 339650278912 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3394266 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3664487 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 339738092544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1247079 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2147187 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17864113984512 trainer/global_step‚499 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4974634 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3664489 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18445297397712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8639123 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18445280358352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18214214821376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4964898 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 $ _timestamp‚1.784497114829097e+09Ô3Š„¦'¨ O D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8629385 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2147184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 349911826432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 338654203176 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 338568056566 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4964898 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18692601842688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2147187 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17864113984512 trainer/global_step‚499 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4974634 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3664489 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18445297397712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8639123 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18445280358352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18214214821376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4964898 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 $ _timestamp‚1.784497114829097e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189010432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚37 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19032235082240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18692584803328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 339650278912 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19032339935232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3394006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18783951600888 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4974634 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17864089080832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 188.741476222 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18214000907264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3664489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19032235082240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 339650278912 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3394266 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3664487 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 339738092544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1247079 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 339330154732 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18250605002752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19070384419328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18729951170560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 340328388608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8656356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211809792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17899784238080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3400783 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4974796 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 340328388608 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3401049 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18821396434122 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1249309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4984546 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18729951170560 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18250393192960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 340416209408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2151477é`Xݱ&««M F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17899784238080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3400783 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4974796 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 340328388608 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3401049 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18821396434122 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1249309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4984546 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18729951170560 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18250393192960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 340416209408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2151477 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2151474 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 339330154732 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18250605002752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19070384419328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18729951170560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 340328388608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8656356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211809792&˜SÌP0¬+ _wandbruntime‚188  _runtime‚1889ùyû­õ 1loss/SRqqt2/assignment_loss‚2.207176252432309 0loss/FRt1/detection_loss‚0.41218304043490994 /loss/FRt2/detection_loss‚1.5009777956232637 % _timestamp‚1.7844971150873642e+09 _runtime‚ 188.99796666 _step‚39 1loss/SRqqt1/assignment_loss‚2.493172255877791 .loss/FBt1/detection_loss‚27.43161605806513 .loss/FBt2/detection_loss‚5.936069709596818 1loss/FBt2/assignment_loss‚0.14245602345714967 epoch‚0 /loss/FRt2/assignment_loss‚4.610962923835306 0loss/FBt1/assignment_loss‚0.3985569056716155 /loss/SRqqt1/detection_loss‚4.46841412859744 1loss/SRqqt2/detection_loss‚0.9229695043425002 %loss/total_loss‚55.04293776195344 trainer/global_step‚499 /loss/FRt1/assignment_loss‚4.5183831640191983Ÿ™ÿ®ù _runtime‚ 188.99796666 _step‚39 1loss/SRqqt1/assignment_loss‚2.493172255877791 .loss/FBt1/detection_loss‚27.43161605806513 .loss/FBt2/detection_loss‚5.936069709596818 1loss/FBt2/assignment_loss‚0.14245602345714967 epoch‚0 /loss/FRt2/assignment_loss‚4.610962923835306 0loss/FBt1/assignment_loss‚0.3985569056716155 /loss/SRqqt1/detection_loss‚4.46841412859744 1loss/SRqqt2/detection_loss‚0.9229695043425002 %loss/total_loss‚55.04293776195344 trainer/global_step‚499 /loss/FRt1/assignment_loss‚4.518383164019198 1loss/SRqqt2/assignment_loss‚2.207176252432309 0loss/FRt1/detection_loss‚0.41218304043490994 /loss/FRt2/detection_loss‚1.5009777956232637 % _timestamp‚1.7844971150873642e+09'L_Sµ%¯‚(:š á‡õÒ£²—+ disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚115.80224609375 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 324.19140625 network.recv‚95276$ disk./.usageGB‚1165.780673980713 disk.nvme0n1.in‚0 disk.md0.out‚ 7.6640625 disk.dm-0.in‚0 cpu‚25.921366064049412) disk./.usagePercent‚75.93425656933482# disk.nvme0n1.out‚839.9541015625$ disk.nvme1n1.out‚115.80224609375 disk.md0.in‚0 disk.dm-1.out‚ 2.84814453125 disk.rbd0.out‚0# memory_percent‚98.0212926864624! proc.memory.rssMB‚ 2775.390625 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚839.9541015625* proc.memory.percent‚16.939640045166016 proc.cpu.threads‚17 disk.dm-0.out‚ 936.78125 network.sent‚480838˜B^Wš°‚(: á‡õÒ¹™ç+ gpu.0.gpu‚78.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.43' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.47666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈfUd0±+ _wandbruntime‚207  _runtime‚207:]IJ¾ & lr-AdamW/pg2‚6.974385516812403e-05 trainer/global_step‚549 % _timestamp‚1.7844971335765216e+09 _runtime‚ 207.487462662 _step‚40 & lr-AdamW/pg1‚6.974385516812403e-05Þhȳ % _timestamp‚1.7844971335765216e+09 _runtime‚ 207.487462662 _step‚40 & lr-AdamW/pg1‚6.974385516812403e-05 & lr-AdamW/pg2‚6.974385516812403e-05 trainer/global_step‚549(xF¹s0´+ _wandbruntime‚207  _runtime‚207ÈÅ,¾¢'µœO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19648846943232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5461650 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 trainer/global_step‚549 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213914112 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚41 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20934491185664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 373588020736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20560920204288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3733525 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3733263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2361684 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 372483678584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 $ _timestamp‚1.784497133577381e+09 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4030539 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20661281278326 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9501925 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20033896537088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9492187 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 385024690176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20661384464296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1371838 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20934596038656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 372397531974 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2361687 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4030539 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20288900785712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20033682622976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 373675834368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20560903164928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 373675834368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20934596038656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1371579 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19648871846912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 384835679744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5461650 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20934491185664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4030537 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5471386 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1371838 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20934596038656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 372397531974 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2361687 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4030539 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20288900785712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20033682622976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 373675834368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20560903164928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 373675834368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20934596038656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1371579 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19648871846912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 384835679744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5461650 RDDeviceStatsMonitor.on_traiŸôU¯Ÿn_batch_start/active_bytes.small_pool.peak‚ 313357824 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20934491185664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4030537 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5471386 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20934491185664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 373588020736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20560920204288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3733525 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3733263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2361684 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 372483678584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 $ _timestamp‚1.784497133577381e+09 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4030539 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20661281278326 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9501925 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20033896537088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9492187 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 385024690176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970)×[2È0·+ _wandbruntime‚207  _runtime‚207DÖÎ~¬&¸¦M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9519158 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4037860 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5481298 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 $ _timestamp‚1.784497133835834e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20598269532160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 374353951232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207615488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚42 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5481298 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5471548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20325772853472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 374353951232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20972640522752 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20598286571520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19684542100480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9509406 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4037858 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19684567004160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5471548 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 374266130432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20598269532160 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385717688832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1374100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 373159630140 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4037858 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3740074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9509406 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20070077077504v­¯E°&¹ªM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5481298 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 $ _timestamp‚1.784497133835834e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20598269532160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 374353951232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207615488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚42 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5481298 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5471548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20325772853472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 374353951232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20972640522752 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20598286571520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19684542100480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9509406 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4037858 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19684567004160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5471548 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 374266130432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20598269532160 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385717688832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1374100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 373159630140 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4037858 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3740074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9509406 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20070077077504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9519158 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4037860 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5968138 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22836747289088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4396587 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22429238565888 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 407613576192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21433604805632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22538817327704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 406313153992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 406227007382 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22836747289088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4072063 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2576187 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22132487134352 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5968138 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22836747289088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4396587 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22429238565888 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 407613576192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21433604805632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22538817327704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 406313153992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 406227007382 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22836747289088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4072063 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2576187 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0-| ×w0Ç+ _wandbruntime‚226  _runtime‚226’FJ(°&ȪM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10372208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1498573 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1498324 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 420598540800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22874896626176 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 408291693056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22466604933120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784497152583424e+09 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21889923407360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4403910 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207615488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21469299962880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5968300 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 408203872256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2580477 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5978050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚46 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5968300 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 408291693056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21889715791872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22874896626176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22466587893760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22169359202112 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4079050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2580474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4403910 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5978050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 408203872256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚ 226.494527268 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 420415828992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22576365347020 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10381960 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4078798 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4403908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10372208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4403908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22576262160938 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22466587893760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10381960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚599 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406902958826 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482X3¹BÉ®M MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4403908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10372208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4403908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22576262160938 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22466587893760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10381960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚599 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406902958826 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10372208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1498573 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1498324 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 420598540800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22874896626176 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 408291693056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22466604933120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784497152583424e+09 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21889923407360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4403910 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207615488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21469299962880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5968300 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 408203872256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2580477 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5978050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚46 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5968300 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 408291693056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21889715791872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved8Ûä5 _bytes.all.current‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22874896626176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22466587893760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22169359202112 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4079050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2580474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4403910 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5978050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 408203872256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚ 226.494527268 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 420415828992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22576365347020 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10381960 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4078798 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482.f×0Ê+ _wandbruntime‚226  _runtime‚226åäÝüËö 1loss/SRqqt2/assignment_loss‚2.170790750472272 /loss/FRt1/detection_loss‚0.3701975693161514 /loss/FBt2/detection_loss‚6.6127911117995195 _runtime‚ 226.495005399 /loss/FRt2/assignment_loss‚4.394228923859908 /loss/FBt1/detection_loss‚15.691055903521677 trainer/global_step‚599 _step‚47 /loss/FRt1/assignment_loss‚4.525807124906473 2loss/SRqqt1/assignment_loss‚1.7275283647006785 .loss/FRt2/detection_loss‚1.300685744596898 %loss/total_loss‚42.31374148928744 /loss/FBt1/assignment_loss‚0.405772112520127 /loss/FBt2/assignment_loss‚0.274245473429222 1loss/SRqqt1/detection_loss‚3.5676135985379123 1loss/SRqqt2/detection_loss‚1.2730248116265952 epoch‚0 % _timestamp‚1.7844971525843372e+09:ˈÌú /loss/FRt1/assignment_loss‚4.525807124906473 2loss/SRqqt1/assignment_loss‚1.7275283647006785 .loss/FRt2/detection_loss‚1.300685744596898 %loss/total_loss‚42.31374148928744 /loss/FBt1/assignment_loss‚0.405772112520127 /loss/FBt2/assignment_loss‚0.274245473429222 1loss/SRqqt1/detection_loss‚3.5676135985379123 1loss/SRqqt2/detection_loss‚1.2730248116265952 epoch‚0 % _timestamp‚1.7844971525843372e+09 1loss/SRqqt2/assignment_loss‚2.170790750472272 /loss/FRt1/detection_loss‚0.3701975693161514 /loss/FBt2/detection_loss‚6.6127911117995195 _runtime‚ 226.495005399 /loss/FRt2/assignment_loss‚4.394228923859908 /loss/FBt1/detection_loss‚15.691055903521677 trainer/global_step‚599 _step‚47/·I ›Í‚(: ŽˆõÒ¾£Ñ+ gpu.0.gpu‚82.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.251' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.75033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷»n΂(: ŽˆõÒ×Þ¹, proc.cpu.threads‚17) disk./.usagePercent‚75.93414699229564% disk.nvme0c0n1.out‚852.8759765625# disk.nvme0n1.out‚852.8759765625" disk.nvme1c1n1.out‚ 117.8984375 disk.nvme0c0n1.in‚0 disk.nvme1n1.out‚ 117.8984375 disk.dm-0.in‚0 disk.dm-0.out‚ 949.109375 disk.dm-1.in‚0$ memory_percent‚98.02160263061523& proc.memory.availableMB‚ 324.140625 cpu‚25.92421062901633 network.recv‚108277 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚8.78125 disk.dm-1.out‚ 3.1396484375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚580227% disk./.usageGB‚1165.7789916992188 disk.nvme0n1.in‚0! proc.memory.rssMB‚ 2775.390625* proc.memory.percent‚16.939640045166016Xª>Â0Ï+ _wandbruntime‚244  _runtime‚244ÉÀÃн _runtime‚ 244.95740466 _step‚48 & lr-AdamW/pg1‚8.244765392370217e-05 & lr-AdamW/pg2‚8.244765392370217e-05 trainer/global_step‚649 % _timestamp‚1.7844971710464735e+098Â3/ÇÑÁ _step‚48 & lr-AdamW/pg1‚8.244765392370217e-05 & lr-AdamW/pg2‚8.244765392370217e-05 trainer/global_step‚649 % _timestamp‚1.7844971710464735e+09 _runtime‚ 244.957404660WtÝ 0Ò+ _wandbruntime‚244  _runtime‚244—Dú¨¦'Ó O ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4762637 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4411437 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23673201224704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24416147005142 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24297556927488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23218362668032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24739108245504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24297539888128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 441463504384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 trainer/global_step‚649 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23218387571712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24739003392512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24739003392512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23976107561712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 441551318016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6464890 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11217791 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2790684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784497171047343e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6455154 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 441551318016 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24739108245504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 440142629400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4762639 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 441463504384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚49 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4762639 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4411182 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6464890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24297556927488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24297539888128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11227529 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24416250191112 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4762637 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23672993602048 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 454813652992 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 440056482790 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784497171047343e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6455154 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 441551318016 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24739108245504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 440142629400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4762639 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 441463504384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚49 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4762639 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4411182 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6464890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24297556927488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24297539888128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11227529 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24416250191112 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4762637 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23672993602048 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 454813652992 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 440056482790 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24739003392512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24739003392512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23976107561712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 441551318016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6464890 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11217791 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2790684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24334923294720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1622977 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2794977 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 455506506240 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4769960 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 442229434880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24334923294720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178517504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 440818580956 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4418214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2794974 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24777152729600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4417951 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6465052 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24012962590112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4769958 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24334906255360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11235010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1623237 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 440732434234 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11244762 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6474802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248ôKʵ&ׯM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6465052 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24012962590112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4769958 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24334906255360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11235010 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1623237 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 440732434234 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11244762 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6474802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 442141614080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚50 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 442229434880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23254082728960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23254057825280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24334923294720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1622977 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2794977 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 455506506240 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4769960 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 442229434880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24334923294720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178517504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 440818580956 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4418214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2794974 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24777152729600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4417951 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚4822ù…à†0Ø+ _wandbruntime‚245  _runtime‚245×¼úÙô 1loss/SRqqt1/assignment_loss‚1.666655784293177 2loss/SRqqt2/assignment_loss‚2.1536699293633186 .loss/FRt2/detection_loss‚1.344546934905686 % _timestamp‚1.7844971713067863e+09 0loss/SRqqt1/detection_loss‚2.831947083537689 .loss/FBt2/detection_loss‚4.374373627452223 %loss/total_loss‚43.43894807862258 epoch‚0 _runtime‚ 245.21741142 _step‚51 /loss/FRt1/assignment_loss‚4.538824478785197 /loss/FRt2/assignment_loss‚4.057105578048319 0loss/FBt1/assignment_loss‚0.2672690622275695 /loss/FRt1/detection_loss‚0.4590596481752589 0loss/SRqqt2/detection_loss‚1.106250331015382 .loss/FBt1/detection_loss‚20.50175277102325 1loss/FBt2/assignment_loss‚0.13749284979550996 trainer/global_step‚649ÄúcþÚø 0loss/SRqqt2/detection_loss‚1.106250331015382 .loss/FBt1/detection_loss‚20.50175277102325 1loss/FBt2/assignment_loss‚0.13749284979550996 trainer/global_step‚649 1loss/SRqqt1/assignment_loss‚1.666655784293177 2loss/SRqqt2/assignment_loss‚2.1536699293633186 .loss/FRt2/detection_loss‚1.344546934905686 % _timestamp‚1.7844971713067863e+09 0loss/SRqqt1/detection_loss‚2.831947083537689 .loss/FBt2/detection_loss‚4.374373627452223 %loss/total_loss‚43.43894807862258 epoch‚0 _runtime‚ 245.21741142 _step‚51 /loss/FRt1/assignment_loss‚4.538824478785197 /loss/FRt2/assignment_loss‚4.057105578048319 0loss/FBt1/assignment_loss‚0.2672690622275695 /loss/FRt1/detection_loss‚0.45905964817525893ãTH¢Û‚(:Ž ˆõÒ¹÷Ø+ disk.dm-1.in‚0 disk.dm-1.out‚ 3.451171875 disk.nvme0c0n1.in‚0$ memory_percent‚98.02236557006836& proc.memory.availableMB‚ 324.015625! proc.memory.rssMB‚ 2775.390625 network.sent‚640218) disk./.usagePercent‚75.93416811258663 disk.md0.out‚10.71875 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 862.763671875 disk.nvme1n1.in‚0 cpu‚25.92767066361469 proc.cpu.threads‚17 network.recv‚113212% disk./.usageGB‚1165.7793159484863 disk.md0.in‚0 disk.dm-0.out‚ 958.58203125 disk.rbd0.out‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 862.763671875! disk.nvme1c1n1.out‚ 121.734375 disk.dm-0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.out‚ 121.734375* proc.memory.percent‚16.9396400451660161E%œÜ‚(:‘ ˆõÒÏ«‚, gpu.0.gpu‚87.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚147.735' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚49.245000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aK0Ý+ _wandbruntime‚263  _runtime‚263—õó²ÄÞ¾ & lr-AdamW/pg2‚8.879955330149125e-05 trainer/global_step‚699 % _timestamp‚1.7844971897781513e+09 _runtime‚ 263.689081814 _step‚52 & lr-AdamW/pg1‚8.879955330149125e-05˜ƒtgÈß _step‚52 & lr-AdamW/pg1‚8.879955330149125e-05 & lr-AdamW/pg2‚8.879955330149125e-05 trainer/global_step‚699 % _timestamp‚1.7844971897781513e+09 _runtime‚ 263.6890818144D“ã”0à+ _wandbruntime‚263  _runtime‚263ê¤'ážO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26641259495936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6951906 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4751000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 489558178304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 263.6933792 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6961642 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 475401246208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1745539 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473885958198 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 475489059840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5128687 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25003120530432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3005187 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3005184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 trainer/global_step‚699 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1745813 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25492886331392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26165858249728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26165875289088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5128689 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26165858249728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330é?4`ïâ¢O OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6961642 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 475401246208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1745539 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473885958198 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 475489059840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5128687 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25003120530432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3005187 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3005184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 trainer/global_step‚699 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1745813 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25492886331392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26165858249728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26165875289088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5128689 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26165858249728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26641259495936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6951906 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4751000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 489558178304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 263.69337925ëŽh/0ã+ _wandbruntime‚263  _runtime‚263c>Ò ±&ä«M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12097812 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25856583017472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚699 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 476167176704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26203241656320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5136010 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3009474 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5136008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1748198 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25529075231744 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6961804 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26203241656320 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 490259544064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12107564 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6961804 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178517504 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4757427 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4757675 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12097812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203421184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26679408833024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6971554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26331231073836 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 476167176704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5136010 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 476079355904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25038840591360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6971554 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25856565978112 _step‚54 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1747953 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚699 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 476167176704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26203241656320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5136010 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3009474 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5136008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1748198 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25529075231744 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6961804 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26203241656320 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 490259544064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12107564 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6961804 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178517504 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4757427 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4757675 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12097812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203421184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26679408833024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6971554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26331231073836 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 476167176704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5136010 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 476079355904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25038840591360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6971554 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25856565978112 _step‚54 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1747953 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚06[®„80æ+ _wandbruntime‚263  _runtime‚263á©N çú trainer/global_step‚699 1loss/SRqqt1/assignment_loss‚1.847641119653462 .loss/FRt2/detection_loss‚1.445442036547459 /loss/FBt1/detection_loss‚20.355420002222267 _step‚55 1loss/FBt2/assignment_loss‚0.22000076017775427 %loss/total_loss‚42.38475356005993 /loss/FRt2/assignment_loss‚4.004376907166236 2loss/SRqqt2/assignment_loss‚1.8682616468454372 1loss/FBt1/assignment_loss‚0.16127897415922401 1loss/SRqqt2/detection_loss‚1.1427160896199962 /loss/FBt2/detection_loss‚2.9254378336393527 % _timestamp‚1.7844971900367653e+09 _runtime‚ 263.947400829 /loss/FRt1/assignment_loss‚4.565040881817158 /loss/FRt1/detection_loss‚0.4169634158101677 1loss/SRqqt1/detection_loss‚3.4321738924014102 epoch‚0Rž'uèþ 1loss/SRqqt1/assignment_loss‚1.847641119653462 .loss/FRt2/detection_loss‚1.445442036547459 /loss/FBt1/detection_loss‚20.355420002222267 _step‚55 1loss/FBt2/assignment_loss‚0.22000076017775427 %loss/total_loss‚42.38475356005993 /loss/FRt2/assignment_loss‚4.004376907166236 2loss/SRqqt2/assignment_loss‚1.8682616468454372 1loss/FBt1/assignment_loss‚0.16127897415922401 1loss/SRqqt2/detection_loss‚1.1427160896199962 /loss/FBt2/detection_loss‚2.9254378336393527 % _timestamp‚1.7844971900367653e+09 _runtime‚ 263.947400829 /loss/FRt1/assignment_loss‚4.565040881817158 /loss/FRt1/detection_loss‚0.4169634158101677 1loss/SRqqt1/detection_loss‚3.4321738924014102 epoch‚0 trainer/global_step‚6997Vb&'é‚(:œ ¬ˆõÒ¾äà+ disk.rbd0.out‚0& disk.nvme0c0n1.out‚864.39404296875& disk.nvme1c1n1.out‚123.15380859375 disk.md0.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚864.39404296875$ memory_percent‚98.02587032318115 cpu‚25.930607870488544 proc.cpu.threads‚17 network.recv‚117605) disk./.usagePercent‚75.93417208817083 disk.nvme1c1n1.in‚0 disk.dm-0.out‚ 959.015625 disk.nvme0c0n1.in‚0( proc.memory.availableMB‚ 323.44140625! proc.memory.rssMB‚ 2775.390625* proc.memory.percent‚16.939640045166016% disk./.usageGB‚1165.7793769836426 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 3.556640625 network.sent‚674191$ disk.nvme1n1.out‚123.15380859375 disk.md0.out‚11.9375 disk.dm-1.in‚0 disk.rbd0.in‚0YÈ°|›ê‚(: ¬ˆõÒâËŒ, gpu.0.gpu‚87.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚200.157' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.71900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÜõ0ë+ _wandbruntime‚282  _runtime‚282ôTÄì¾ & lr-AdamW/pg1‚9.515145267928032e-05 & lr-AdamW/pg2‚9.515145267928032e-05 trainer/global_step‚749 % _timestamp‚1.7844972085609276e+09 _runtime‚ 282.473609011 _step‚56ÊNrŸÈí & lr-AdamW/pg1‚9.515145267928032e-05 & lr-AdamW/pg2‚9.515145267928032e-05 trainer/global_step‚749 % _timestamp‚1.7844972085609276e+09 _runtime‚ 282.473609011 _step‚568Ö/0â0î+ _wandbruntime‚282  _runtime‚282g±ƒmï¡O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1869290 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27663297298352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494739 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7448658 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28034176611328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7458394 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚ 282.474147151 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 509426801664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 507801580216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28171115917928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚749 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12953133 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494739 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28543515599360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28171012731958 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7458394 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494737 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5088716 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1869032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26787903296512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28034176611328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 507715433606 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28543515599360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213914112 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28034193650688 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28543620452352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12943395 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 509338988032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3219687 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28543620452352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28034193650688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3219684¿nÁE«'ð¥O ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 509426801664 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7448658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26787878392832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28543515599360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213914112 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28034193650688 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28543620452352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12943395 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 509338988032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3219687 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28543620452352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28034193650688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3219684 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1869290 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27663297298352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494739 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7448658 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28034176611328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7458394 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚ 282.474147151 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 509426801664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 507801580216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28171115917928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚749 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12953133 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494739 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28543515599360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28171012731958 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7458394 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494737 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5088716 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1869032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26787903296512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28034176611328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 507715433606 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 510017097728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28071560017920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5095398 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1871424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5502060 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12970366 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27348956439040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7468306 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26823573550080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28071542978560 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 508477531772 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚58 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12960614 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7468306 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5502060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28208663937244 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 508391385050 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7458556 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 510104918528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844972088199415e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7458556 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3223974 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27700169366112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12970366 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 510017097728 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28071542978560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28581664936448 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27348755115008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5502058 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5502058 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1871667 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 510104918528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28208560751162 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _runtime‚ 282.731033319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3223977 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0©f‰Ló¯M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12960614 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7468306 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5502060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28208663937244 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 508391385050 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7458556 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 510104918528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844972088199415e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7458556 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3223974 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27700169366112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12970366 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 510017097728 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28071542978560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28581664936448 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27348755115008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5502058 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5502058 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1871667 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 510104918528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28208560751162 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _runtime‚ 282.731033319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3223977 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712¨lS®ž  LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27700186405472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 525357985280 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 510017097728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28071560017920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5095398 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1871424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5502060 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12970366 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27348956439040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7468306 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26823573550080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28071542978560 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 508477531772 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚58 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29902494972928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180621824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30048548781336 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7945410 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1993586 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30445771702784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 541544909014 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7955146 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30048445595366 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7945410 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5427522 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3434184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5860789 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559434060288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30445771702784 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 543364543488 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5860787 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28572636255232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29506900686352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29902512012288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0HEœfª'€¤O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 543364543488 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 543276729856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5860787 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 543276729856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5427773 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5860787 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28572636255232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29506900686352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29902512012288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29902494972928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180621824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30048548781336 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7945410 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1993586 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30445771702784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 541544909014 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7955146 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30048445595366 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7945410 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5427522 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3434184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5860789 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559434060288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30445771702784 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 543364543488 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3438474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29543772754112 _step‚62 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 542307007180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13823416 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 543954839552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5434513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7965058 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30483921039872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13823416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7965058 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 195294720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30086096800652 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29168455988224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚799 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30483816179712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 544042660352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560319870464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 609417PE”gØ9591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29939878379520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 542220860458 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220198400 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7844972275361965e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 543954839552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1995784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5868110 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7955308 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29543789793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13833168 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 544042660352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5434258 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30085993614570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13833168 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A„i³&ƒ­M WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 543954839552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1995784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5868110 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7955308 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29543789793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13833168 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 544042660352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5434258 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30085993614570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13833168 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3438474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29543772754112 _step‚62 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 542307007180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13823416 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 543954839552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5434513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7965058 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30483921039872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13823416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7965058 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 195294720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30086096800652 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29168455988224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚799 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30483816179712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 544042660352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560319870464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29939878379520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 542220860458 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220198400 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7844972275361965e+09>p>}^0„+ _wandbruntime‚301  _runtime‚301Ž¨>……û %loss/total_loss‚41.81646022744965 1loss/FBt1/assignment_loss‚0.12908648619010749 1loss/FBt2/assignment_loss‚0.18519073309289524 0loss/SRqqt1/detection_loss‚3.837543315166645 /loss/FBt1/detection_loss‚18.433510610456413 trainer/global_step‚799 /loss/FRt1/assignment_loss‚4.082331471972996 /loss/FRt2/detection_loss‚1.3196551423045542 epoch‚0 % _timestamp‚1.7844972275375566e+09 _step‚63 .loss/FBt2/detection_loss‚5.643219831392439 _runtime‚ 301.448197412 0loss/FRt2/assignment_loss‚3.3722183211096404 2loss/SRqqt1/assignment_loss‚1.7102798719534138 1loss/SRqqt2/assignment_loss‚1.648382506971993 0loss/FRt1/detection_loss‚0.33898132223439364 1loss/SRqqt2/detection_loss‚1.1160606146041652ý¡öj†ÿ 1loss/FBt2/assignment_loss‚0.18519073309289524 0loss/SRqqt1/detection_loss‚3.837543315166645 /loss/FBt1/detection_loss‚18.433510610456413 trainer/global_step‚799 /loss/FRt1/assignment_loss‚4.082331471972996 /loss/FRt2/detection_loss‚1.3196551423045542 epoch‚0 % _timestamp‚1.7844972275375566e+09 _step‚63 .loss/FBt2/detection_loss‚5.643219831392439 _runtime‚ 301.448197412 0loss/FRt2/assignment_loss‚3.3722183211096404 2loss/SRqqt1/assignment_loss‚1.7102798719534138 1loss/SRqqt2/assignment_loss‚1.648382506971993 0loss/FRt1/detection_loss‚0.33898132223439364 1loss/SRqqt2/detection_loss‚1.1160606146041652 %loss/total_loss‚41.81646022744965 1loss/FBt1/assignment_loss‚0.12908648619010749?S2eÈ(‡‚(: ÙˆõÒ袾+ disk.md0.in‚0$ memory_percent‚98.29201698303223* proc.memory.percent‚17.205429077148438 network.sent‚772356 disk.nvme0c0n1.in‚0 disk.md0.out‚ 13.41796875 proc.memory.rssMB‚ 2818.9375 cpu‚26.066655350858035 proc.cpu.threads‚17) disk./.usagePercent‚75.93136532573392# disk./.usageGB‚1165.73628616333 disk.dm-1.in‚0$ disk.nvme1n1.out‚125.88330078125 disk.dm-0.in‚0 disk.dm-0.out‚ 962.62109375 network.recv‚129971 disk.rbd0.out‚0& disk.nvme0c0n1.out‚869.01806640625 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 279.8359375 disk.dm-1.out‚ 4.126953125 disk.rbd0.in‚0$ disk.nvme0n1.out‚869.01806640625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚125.88330078125A-Ý›ˆ‚(: ÙˆõÒãÍ+ gpu.0.gpu‚50.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚189.635' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.211666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cûûH‰bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÀ!¿.JŠbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP ¥ÞJ@‹b; particle/accuracy_1_of_2"trainer/global_step2:JPÎ?ùgHŒbC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPdå;b6 jet/accuracy_3_of_5"trainer/global_step2:JPƇ`HNŽbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP-1-ÚJbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÌØæNbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP·[žH‘bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¹ ÏN’bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP¤–ȹN“bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÜ Ð{J”bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP ^H•bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPòƒ~ÇJ–bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPeA@—b; particle/accuracy_2_of_5"trainer/global_step2:JP{©ÝìN˜bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPm"ÓPN™bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPúMÅëNšbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP*–J›bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÞ\•T;œb6 jet/accuracy_2_of_4"trainer/global_step2:JPkY-HbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÇ彋NžbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP?¬3JŸbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP1Î-ÐH bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPh#æ;¡b6 jet/accuracy_3_of_4"trainer/global_step2:JPÒ ÛÐ9¢b4 particle/accuracy"trainer/global_step2:JPCϪ@£b; particle/accuracy_5_of_5"trainer/global_step2:JP~°IJJ¤bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPï ÔJ¥bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÔ³aåH¦bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPñUJ§bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP‘ƒ˜öN¨bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP²‘Y~H©bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÞ“¦ãJªbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPV‡$<«b7 particle/sensitivity"trainer/global_step2:JP"Õ”H¬bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPs7#»H­bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPhÞ$H®bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP…lTJ¯bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP šCÜH°bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP-çЮN±bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPô©56N²bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP· rJ³bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâž6 J´bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP©vkäHµbC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP+ï€@N¶bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÖîxH·bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP#ˆ¾¥H¸bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¾`ÉJ¹bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP:xZ)JºbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP®‚H»bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP4£¿¦J¼bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPoà]±J½bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPµkVSJ¾bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPc¥÷<¿b7 particle/specificity"trainer/global_step2:JPt_PNÀbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÏQÏHÁbC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÑîTâHÂbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÄ¢ JÃbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¹[,ŸHÄbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÉB;Åb6 jet/accuracy_5_of_5"trainer/global_step2:JPEVNÆbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPxjçµJÇbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPk},HÈbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPh`Z¤JÉbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPC›e¤NÊbI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP-Þ]ÆJËbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰™DæHÌbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPìoùãJÍbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPìxNÎbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP8¸>öJÏbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¥@tB;Ðb6 jet/accuracy_2_of_2"trainer/global_step2:JPLx‰ HÑbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÒš™JÒbE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPä }¥JÓbE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP·PÓNÔbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP &\Ü;Õb6 jet/accuracy_2_of_3"trainer/global_step2:JPOàóHÖbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÃÆ:!J×bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPq©HJØbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPØ÷œNÙbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPpãb+JÚbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP·<~HÛbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPœ­¦”JÜbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP($¬hNÝbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPǨE[JÞbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPšE´HßbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP3›„xNàbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÊópNábI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP²AÊJâbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÜTNãbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP6KHäbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP‡`æ~JåbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPùñ5HæbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPD½õHçbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPúû‚JèbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPLxy;éb6 jet/accuracy_1_of_2"trainer/global_step2:JP8±zHêbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP=[NëbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÄ ˜žNìbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPº«ÁHíbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPjÜGîbB validation_average_jet_accuracy"trainer/global_step2:JPú,ºŠ@ïb; particle/accuracy_3_of_5"trainer/global_step2:JPG©‹–HðbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¶n¶ÒHñbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¿Õ56HòbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP ’p¶@ób; particle/accuracy_1_of_3"trainer/global_step2:JPÙ+ÏHôbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPòþÊHõbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÖÀJöbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP >‰éH÷bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP_ÏÂNøbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP0„ªÙ†N–bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPØ¥ë&J—bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP‰À;˜b6 jet/accuracy_4_of_4"trainer/global_step2:JP0vó®H™bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP—×LPJšbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP/-N›bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPh¹—°NœbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP(׈JbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÜ—cLJžbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPhæ­JŸbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPì_îˆH bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP'„ªÆN¡bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÀ ÀN¢bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÉÀN£bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP$lá†H¤bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPYïJ¥bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPAgп@¦b; particle/accuracy_4_of_4"trainer/global_step2:JP«\áË@§b; particle/accuracy_4_of_5"trainer/global_step2:JPQU²¹H¨bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP ®˜:J©bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÊFxãNªbI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP,W’NH«bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP¿‚J¬bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP'y²©H­bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPs8ÕJ®bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPpv6ŠJ¯bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPV×ÖÞN°bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPL'QrJ±bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPks¼N²bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP«ÖŽN³bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPb–Ø@´b; particle/accuracy_3_of_3"trainer/global_step2:JP#^âNµbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPåÈ5N¶bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPoq-*H·bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPW“§×J¸bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÂþ¤éJ¹bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPo¢’HºbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPBâ¼@J»bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP²w3pH¼bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP»ZWjN½bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÂéÜNJ¾bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP†Å+¨J¿bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP"ˆ8_HÀbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¥i`SJÁbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÐã[ÇHÂbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP€k0 JÃbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPí? ÄbE "PurÆÐf@ity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPéÖå7JÅbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÍ©?HÆbC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP•ÁõòJÇbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÿ Ð*HÈbC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP«‹¨¤HÉbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPª7}ŠNÊbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPµ‰ÚþHËbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP7“ß$HÌbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP¨©÷ÉJÍbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP C JÎbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP—ˆÆ;Ïb6 jet/accuracy_2_of_5"trainer/global_step2:JPHz{ÕHÐbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP6¨ NÑbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPݦbøHÒbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP]’uŸ8Ób3 particle/f_score"trainer/global_step2:JP“¥ÞÀJÔbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÈÝþJÕbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPG3@pHÖbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPûnH6N×bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPè~ExHØbC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP¥EmâNÙbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP8šžTJÚbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPmŽÖNÛbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP63/;Üb6 jet/accuracy_1_of_1"trainer/global_step2:JPRy¼mHÝbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP71HÞbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP}£°NJßbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPgúNàbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP—JbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP‚Œ|ïN‚bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÈĨOHƒbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP ·ŒJ„bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPCBJ…bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPRO¿ñJ†bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÆÓ€bJ‡bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP'(^–NˆbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP_LhT@‰b; particle/accuracy_2_of_4"trainer/global_step2:JP’ø4ë@Šb; particle/accuracy_3_of_4"trainer/global_step2:JP’ígH‹bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÖ‹À.NŒbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPrŽ[hJbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPs¦jòNŽbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP•©÷²HbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP×"/7HbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPŠì_2H‘bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP°û¬‘J’bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP´3ÉJ“bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÌð&ÒN”bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPñLæÕJ•bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPP|î{H–bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP¤ZDN—bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP˵ ÐJ˜bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÇrœÓH™bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP-`,HšbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPaóâùN›bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP›0íHœbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPg°0éJbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPj28JžbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPn¦]éNŸbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPh_³°J bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP,/³FJ¡bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¤¿%¢N¢bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPV8 Ä;£b6 jet/accuracy_1_of_3"trainer/global_step2:JPÕUŸ;¤b6 jet/accuracy_4_of_5"trainer/global_step2:JPûýûJ¥bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPŽÙ)nJ¦bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPWZÖKJ§bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP´øRJ¨bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPGˆ 0©+ _wandbruntime‚317  _runtime‚317gõ¹I=ªÃz ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9758771657943726 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 jet/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.02162303775548935 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.08516848832368851 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.25834956765174866 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11786626279354095 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2867754399776459 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1587173491716385 *jet/accuracy_2_of_2‚0.0486849769949913 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.26277613639831543 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2396136224269867 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1261092871427536 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1594180464744568 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3282030522823334 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07098164409399033 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.35465431213378906 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.47999998927116394 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.028861738741397858 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8935686945915222 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9722506403923035 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.023585686460137367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968112707138062 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.08978953212499619 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2826537489891052 +jet/accuracy_1_of_2‚0.45846879482269287 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14038899540901184 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18592539429664612 5validation_average_jet_accuracy‚0.164895698428154 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9776055216789246 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9943665862083435 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07383676618337631 particle/accuracy_1_of_3‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.16371162235736847 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.24038167297840118 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9752593040466309 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.001070226076990366 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.15078464150428772 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.23216941952705383 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.19429823756217957 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06819681078195572 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14868420362472534 % _timestamp‚1.7844972431396472e+09 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9900880455970764 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.022693654522299767 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _step‚64 )loss/val_total_loss‚39.50231918207898 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0010948354611173272 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.5303776264190674 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.22833901643753052 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_4‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00176609109621495 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.005953060928732157 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005970935802906752 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.37065982818603516 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.024040790274739265 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.15078464150428772 /particle/accuracy_4_of_4‚0.9939531683921814 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966225624084473 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.24312812089920044 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06630656868219376 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966831207275391 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7404792904853821 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0028218207880854607 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9954123497009277 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_3_of_3‚0.9986624121665955 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.14059828221797943 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.761695146560669 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952649474143982 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.19696968793869019 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.004659280646592379 & Purity/2FRt1SRqqt1FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07149175554513931 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07268886268138885 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.33809423446655273 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9867231249809265 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 trainer/global_step‚800 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14035740494728088 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1863105297088623 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2408391833305359 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.22182562947273254 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.0728427916765213 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.5299129486083984 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.031233396381139755 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9889979362487793 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.25441819429397583 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_1‚0.1318080872297287 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914640784263611 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.36511197686195374 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9907599091529846 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13889606297016144 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.07605903595685959 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1026528999209404 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3220963180065155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 epoch‚0 /particle/accuracy_2_of_3‚0.9996549487113953 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.31015273928642273 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9889942407608032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.014190899208188057 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17491041123867035 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0019480159971863031 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.20454545319080353 particle/accuracy_1_of_5‚1 _runtime‚ 317.051004876 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.13108669221401215 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9951818585395813 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.02927272394299507 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9937258362770081 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06833041459321976 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.031767457723617554 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00176609109621495 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13855700194835663 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2474014163017273 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2072649747133255 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002804212737828493 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9954123497009277 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.156269833445549 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11516095697879791 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0717007964849472 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11394019424915314 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9513792991638184 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963117241859436 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.21296297013759613 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2583344876766205 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5601048469543457 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18076056241989136 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_3‚0.5541455745697021 jet/accuracy_4_of_5‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.765045166015625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11154760420322418 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.02810031548142433 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.21891814470291138 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.19123747944831848 particle/accuracy_1_of_2‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1598726063966751 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0729166641831398 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.161209836602211 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.18830890953540802 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 particle/accuracy_2_of_5‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.14141036570072174 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.1847117692232132 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.24312812089920044 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10480941087007523 ,jet/accuracy_3_of_4‚0.007142857648432255 (particle/accuracy‚0.7739917039871216 particle/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15879355370998383 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18763501942157745 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2777777910232544 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.14064456522464752 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.5931578874588013 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1565779149532318 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.0299479179084301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11516095697879791 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.18863549828529358 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.23630307614803314 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.031156573444604874 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.02927272394299507 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17596761882305145 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9756897687911987 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.16183671355247498 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.47999998927116394 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6908432841300964 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.8234471082687378¥¿± #«Çz <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0019480159971863031 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.20454545319080353 particle/accuracy_1_of_5‚1 _runtime‚ 317.051004876 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.13108669221401215 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9951818585395813 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.02927272394299507 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9937258362770081 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06833041459321976 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.031767457723617554 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00176609109621495 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13855700194835663 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2474014163017273 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2072649747133255 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002804212737828493 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9954123497009277 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.156269833445549 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11516095697879791 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0717007964849472 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11394019424915314 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9513792991638184 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963117241859436 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.21296297013759613 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2583344876766205 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5601048469543457 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18076056241989136 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_3‚0.5541455745697021 jet/accuracy_4_of_5‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.765045166015625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11154760420322418 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.02810031548142433 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.21891814470291138 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.19123747944831848 particle/accuracy_1_of_2‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1598726063966751 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0729166641831398 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.161209836602211 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.18830890953540802 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 particle/accuracy_2_of_5‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.14141036570072174 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.1847117692232132 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.24312812089920044 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10480941087007523 ,jet/accuracy_3_of_4‚0.007142857648432255 (particle/accuracy‚0.7739917039871216 particle/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15879355370998383 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18763501942157745 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2777777910232544 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.14064456522464752 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.5931578874588013 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1565779149532318 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.0299479179084301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11516095697879791 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.18863549828529358 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.23630307614803314 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.031156573444604874 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.02927272394299507 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17596761882305145 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9756897687911987 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.16183671355247498 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.47999998927116394 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6908432841300964 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.8234471082687378 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9758771657943726 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 jet/accuracy_5_of_5‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.02162303775548935 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.08516848832368851 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.25834956765174866 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11786626279354095 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2867754399776459 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1587173491716385 *jet/accuracy_2_of_2‚0.0486849769949913 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.26277613639831543 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2396136224269867 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1261092871427536 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1594180464744568 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3282030522823334 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07098164409399033 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.35465431213378906 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.47999998927116394 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.028861738741397858 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8935686945915222 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9722506403923035 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.023585686460137367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968112707138062 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.08978953212499619 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2826537489891052 +jet/accuracy_1_of_2‚0.45846879482269287 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14038899540901184 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18592539429664612 5validation_average_jet_accuracy‚0.164895698428154 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9776055216789246 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9943665862083435 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07383676618337631 particle/accuracy_1_of_3‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.16371162235736847 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.24038167297840118 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9752593040466309 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.001070226076990366 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.15078464150428772 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.23216941952705383ù²^C @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.19429823756217957 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06819681078195572 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14868420362472534 % _timestamp‚1.7844972431396472e+09 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9900880455970764 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.022693654522299767 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _step‚64 )loss/val_total_loss‚39.50231918207898 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0010948354611173272 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.5303776264190674 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.22833901643753052 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_4‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00176609109621495 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.005953060928732157 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005970935802906752 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.37065982818603516 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.024040790274739265 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.15078464150428772 /particle/accuracy_4_of_4‚0.9939531683921814 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966225624084473 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.24312812089920044 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06630656868219376 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9966831207275391 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7404792904853821 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0028218207880854607 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9954123497009277 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_3_of_3‚0.9986624121665955 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.14059828221797943 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.761695146560669 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952649474143982 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.19696968793869019 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.004659280646592379 & Purity/2FRt1SRqqt1FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07149175554513931 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07268886268138885 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.33809423446655273 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9867231249809265 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 trainer/global_step‚800 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14035740494728088 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1863105297088623 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2408391833305359 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.22182562947273254 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.0728427916765213 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.5299129486083984 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.031233396381139755 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9889979362487793 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.25441819429397583 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_1‚0.1318080872297287 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914640784263611 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.36511197686195374 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9907599091529846 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.13889606297016144 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.07605903595685959 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1026528999209404 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3220963180065155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 epoch‚0 /particle/accuracy_2_of_3‚0.9996549487113953 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.31015273928642273 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9889942407608032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.014190899208188057 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17491041123867035 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688@zm÷·¬‚(: èˆõÒ†·+) disk./.usagePercent‚75.93952645465035# disk.nvme1n1.out‚253.9736328125 disk.dm-0.in‚0 disk.dm-0.out‚963.125 disk.rbd0.in‚0% disk./.usageGB‚1165.8615798950195% disk.nvme1c1n1.out‚253.9736328125 disk.dm-1.in‚0 disk.rbd0.out‚0' proc.memory.percent‚17.218017578125 cpu‚26.076137358946443 network.sent‚819296 disk.nvme0n1.in‚0# disk.nvme0n1.out‚869.7294921875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 150.0546875 proc.memory.rssMB‚2821 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚869.7294921875 disk.dm-1.out‚ 131.900390625$ memory_percent‚99.08413887023926 disk.md0.in‚0 disk.md0.out‚ 13.66015625 network.recv‚134395„âÙœ­‚(:‘ èˆõÒÁ­Ê+ gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚184.959' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚61.653000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îÍü¦0®+ _wandbruntime‚335  _runtime‚335ð'į¾ trainer/global_step‚849 % _timestamp‚1.7844972620602965e+09 _runtime‚ 335.9723874 _step‚65 ' lr-AdamW/pg1‚0.00010175705251429805 ' lr-AdamW/pg2‚0.00010175705251429805¹|á¦È°Â _runtime‚ 335.9723874 _step‚65 ' lr-AdamW/pg1‚0.00010175705251429805 ' lr-AdamW/pg2‚0.00010175705251429805 trainer/global_step‚849 % _timestamp‚1.7844972620602965e+09A+í¿0±+ _wandbruntime‚335  _runtime‚335잊©'²£O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32743493968896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31282817381376 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6464307 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 597634638336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31897906062336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 595854539289 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚66 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 597722659328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8619275 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8619275 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6464305 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32743476929536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3742764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609600 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 595768390823 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33341111567872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 597634638336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31282842285056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5937311 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8609134 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32914865121353 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 _runtime‚ 335.97244727 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33341111567872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5937060 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31897694452736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8609134 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 597722659328 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33341216628224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 $ _timestamp‚1.784497262060967e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 597634638336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31897906062336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 595854539289 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚66 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 597722659328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8619275 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8619275 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6464305 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32743476929536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3742764 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609600 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 595768390823 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33341111567872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 597634638336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31282842285056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5937311 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8609134 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32914865121353 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 _runtime‚ 335.97244727 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33341111567872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5937060 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31897694452736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8609134 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 597722659328 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33341216628224 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 $ _timestamp‚1.784497262060967e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15100815 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 $ _timestamp‚1.784497262318666e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31318512538624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32780860336128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32355865610464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33379261112320 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33379156044800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 598400776192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 596530490845 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31934296483328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32355882649824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15090658 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 trainer/global_step‚849 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocatež>{;md‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32780843296768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3747054 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2196771 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5943825 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3747057 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 615584925184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 598312748032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15100815 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6471626 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8629187 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15090658 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8619032 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 615759041024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32780843296768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33379261112320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31318537442304 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32952309952731 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6471628 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6471628 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32780860336128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚67 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 596444342267 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8619032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 598400776192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31934097463808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840‘ÜÀ¸&¶²M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31934296483328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32355882649824 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15090658 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 trainer/global_step‚849 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32780843296768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3747054 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2196771 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5943825 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3747057 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 615584925184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 598312748032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15100815 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6471626 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8629187 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15090658 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8619032 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 615759041024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32780843296768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33379261112320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31318537442304 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32952309952731 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6471628 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6471628 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32780860336128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚67 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 596444342267 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8619032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 598400776192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31934097463808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 598312748032 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15100815 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 $ _timestamp‚1.784497262318666e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31318512538624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32780860336128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32355865610464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33379261112320 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33379156044800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 598400776192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 596530490845C'ïÕ)0·+ _wandbruntime‚336  _runtime‚336/º[~¸û 2loss/SRqqt1/assignment_loss‚1.6782228501549192 0loss/FBt1/assignment_loss‚0.1011653023914815 1loss/FBt2/assignment_loss‚0.19374253751117398 /loss/FRt2/detection_loss‚1.3745140748848388 0loss/SRqqt2/detection_loss‚0.956144600639887 0loss/FRt1/assignment_loss‚3.9809364904713695 2loss/SRqqt2/assignment_loss‚1.5732605838246885 .loss/FBt2/detection_loss‚6.642926177274983 %loss/total_loss‚38.38014970294274 trainer/global_step‚849 0loss/FRt2/assignment_loss‚3.2838388689152604 1loss/SRqqt1/detection_loss‚3.4635815637091256 /loss/FBt1/detection_loss‚14.762705388549417 epoch‚1 _step‚68 /loss/FRt1/detection_loss‚0.3691112646155897 % _timestamp‚1.7844972623196642e+09 _runtime‚ 336.230306846¬qX~¹ÿ 2loss/SRqqt1/assignment_loss‚1.6782228501549192 0loss/FBt1/assignment_loss‚0.1011653023914815 1loss/FBt2/assignment_loss‚0.19374253751117398 /loss/FRt2/detection_loss‚1.3745140748848388 0loss/SRqqt2/detection_loss‚0.956144600639887 0loss/FRt1/assignment_loss‚3.9809364904713695 2loss/SRqqt2/assignment_loss‚1.5732605838246885 .loss/FBt2/detection_loss‚6.642926177274983 %loss/total_loss‚38.38014970294274 trainer/global_step‚849 0loss/FRt2/assignment_loss‚3.2838388689152604 1loss/SRqqt1/detection_loss‚3.4635815637091256 /loss/FBt1/detection_loss‚14.762705388549417 epoch‚1 _step‚68 /loss/FRt1/detection_loss‚0.3691112646155897 % _timestamp‚1.7844972623196642e+09 _runtime‚ 336.230306846D Ä$P%º‚(:š ÷ˆõÒ¬ò+ disk.rbd0.out‚0 disk.nvme0n1.in‚0' proc.memory.percent‚17.218017578125 disk.dm-0.out‚970.5341796875& disk.nvme1c1n1.out‚254.72900390625 disk.md0.in‚0 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚877.36572265625$ disk.nvme0n1.out‚877.36572265625 cpu‚26.069386991935655& proc.memory.availableMB‚ 149.890625 proc.memory.rssMB‚2821 proc.cpu.threads‚17 disk.dm-1.in‚0 network.sent‚889341 network.recv‚139415 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚254.72900390625 disk.rbd0.in‚0$ memory_percent‚99.08514022827148) disk./.usagePercent‚75.93949912250906% disk./.usageGB‚1165.8611602783203 disk.md0.out‚ 14.0703125 disk.dm-1.out‚131.9990234375åŒÝ¥š»‚(: ÷ˆõÒœ£, gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚200.287' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.76233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ø·'0¼+ _wandbruntime‚354  _runtime‚354ejÿbĽ¾ % _timestamp‚1.7844972805724843e+09 _runtime‚ 354.485400178 _step‚69 & lr-AdamW/pg1‚0.0001017558627284955 & lr-AdamW/pg2‚0.0001017558627284955 trainer/global_step‚899`.^ÑȾ % _timestamp‚1.7844972805724843e+09 _runtime‚ 354.485400178 _step‚69 & lr-AdamW/pg1‚0.0001017558627284955 & lr-AdamW/pg2‚0.0001017558627284955 trainer/global_step‚899Eå<œ0¿+ _wandbruntime‚354  _runtime‚354%Jõ«'À¥O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 629684014697 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9116027 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6274656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 650092731392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2317141 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3957264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35243367671296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34611812330496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 631660401152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6274405 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2317389 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33067575243776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34611795291136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33067600147456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188803072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚70 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34792297984761 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9105886 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34162613970064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9105886 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35243472731648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 631572380160 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 649903928320 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 629597866231Òoýkô Á©O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 631572380160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚899 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35243367671296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34611812330496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 631660401152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6274405 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2317389 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33067575243776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.pef#»»ak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34611795291136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33067600147456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188803072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚70 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34792297984761 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9105886 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34162613970064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9105886 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35243472731648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 631572380160 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 649903928320 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 629597866231 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 629684014697 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9116027 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6274656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 650092731392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2317141 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3957264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6837678 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚71 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15953460 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6837676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33753880860160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9125939 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 632338518016 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34649178697728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3961557 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35281517215744 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15963617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33103270401024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9115784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15953460 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34829846004077 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9125939 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35281412148224 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 650784574976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844972808300672e+09 _runtime‚ 354.74114291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2319659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34649161658368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 632250489856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6281213 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 630273817675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 632338518016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¨*C¸&IJM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33753880860160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9125939 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 632338518016 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34649178697728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3961557 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35281517215744 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15963617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33103270401024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9115784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15953460 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34829846004077 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9125939 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35281412148224 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 650784574976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844972808300672e+09 _runtime‚ 354.74114291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2319659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34649161658368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 632250489856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6281213 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 630273817675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 632338518016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6837678 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚71 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15953460 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6837676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152GûT0Å+ _wandbruntime‚354  _runtime‚354¯ù°Æü 2loss/SRqqt1/assignment_loss‚1.7338476928765187 1loss/SRqqt1/detection_loss‚2.9785285624427984 trainer/global_step‚899 % _timestamp‚1.7844972808309853e+09 /loss/FRt2/assignment_loss‚3.177146135113347 1loss/SRqqt2/detection_loss‚0.9150087213038874 _runtime‚ 354.741707461 _step‚72 epoch‚1 0loss/FRt1/assignment_loss‚4.0070346117019655 2loss/SRqqt2/assignment_loss‚1.4527246590292149 2loss/FBt1/assignment_loss‚0.055366886385551574 .loss/FBt1/detection_loss‚19.20175099720304 .loss/FBt2/detection_loss‚4.418953971861733 %loss/total_loss‚39.90494013245743 1loss/FBt2/assignment_loss‚0.19874626143266344 /loss/FRt1/detection_loss‚0.5283444387256584 /loss/FRt2/detection_loss‚1.2374871943810553ezsÇ€ 2loss/SRqqt1/assignment_loss‚1.7338476928765187 1loss/SRqqt1/detection_loss‚2.9785285624427984 trainer/global_step‚899 % _timestamp‚1.7844972808309853e+09 /loss/FRt2/assignment_loss‚3.177146135113347 1loss/SRqqt2/detection_loss‚0.9150087213038874 _runtime‚ 354.741707461 _step‚72 epoch‚1 0loss/FRt1/assignment_loss‚4.0070346117019655 2loss/SRqqt2/assignment_loss‚1.4527246590292149 2loss/FBt1/assignment_loss‚0.055366886385551574 .loss/FBt1/detection_loss‚19.20175099720304 .loss/FBt2/detection_loss‚4.418953971861733 %loss/total_loss‚39.90494013245743 1loss/FBt2/assignment_loss‚0.19874626143266344 /loss/FRt1/detection_loss‚0.5283444387256584 /loss/FRt2/detection_loss‚1.2374871943810553H<í|É!È‚(:– †‰õÒîï+ cpu‚26.06525993340549 network.recv‚143977 disk.nvme1n1.in‚0 disk.dm-1.out‚132.1396484375 disk.nvme0c0n1.in‚0& proc.memory.availableMB‚ 149.796875 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚255.2412109375 disk.rbd0.out‚0% disk.nvme0c0n1.out‚878.0888671875# disk.nvme0n1.out‚878.0888671875 proc.cpu.threads‚17% disk./.usageGB‚1165.8654670715332 disk.md0.in‚0 disk.md0.out‚ 14.32421875 disk.dm-0.in‚0 disk.dm-0.out‚971.0576171875 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚2821 network.sent‚951647) disk./.usagePercent‚75.93977964966834# disk.nvme1n1.out‚255.2412109375 disk.nvme0n1.in‚0$ memory_percent‚99.08571243286133' proc.memory.percent‚17.218017578125žçŽÉ‚(:ƒ †‰õÒú¹ , gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.768' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Š@0Ê+ _wandbruntime‚373  _runtime‚373¨P<™ÆËÀ _runtime‚ 373.330550897 _step‚73 ' lr-AdamW/pg1‚0.00010175385804188858 ' lr-AdamW/pg2‚0.00010175385804188858 trainer/global_step‚949 % _timestamp‚1.7844972994195974e+09é3 ÊÌÄ ' lr-AdamW/pg1‚0.00010175385804188858 ' lr-AdamW/pg2‚0.00010175385804188858 trainer/global_step‚949 % _timestamp‚1.7844972994195974e+09 _runtime‚ 373.330550897 _step‚73ID¦”…0Í+ _wandbruntime‚373  _runtime‚373ÆKmp ΤO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7196405 JDDeviceStatsMonitor.on_train_batch_start/“Çm.:oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚ 373.331261688 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9612779 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4171764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34852333106176 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 665510121984 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6611918 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35537393534464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9612779 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37145623774720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16809186 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16799043 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7196407 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203220992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36669730848169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36480113652736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37145728835072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9602638 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2439921 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35537190313472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37145623774720 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 663513490105 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6611685 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36006200318704 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9602638 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 684857207296 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7196405 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36480113652736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685035524608 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 663427341639 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7196405 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚ 373.331261688 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9612779 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4171764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34852333106176 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 665510121984 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6611918 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35537393534464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9612779 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37145623774720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16809186 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16799043 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7196407 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203220992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36669730848169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36480113652736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37145728835072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9602638 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2439921 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35537190313472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37145623774720 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 663513490105 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362JKxwN0Ð+ _wandbruntime‚373  _runtime‚373r¸_X³&Ñ­M MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9622691 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16816262 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 685551937536 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180407296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16826419 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6618650 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34888053167104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36517480019968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 664103293083 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7203726 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 666276259840 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9622691 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9612536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6618393 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36517497059328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2442593 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183773319168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183773319168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4176057 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 $ _timestamp‚1.784497299676768e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36707175679547 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7203726 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35573785511936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 666188231680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9612536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7203728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2442339 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 664189441661 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚75 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 _runtime‚ 373.58789639 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 666188231680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16826419 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4176054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36043072386464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16816262 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 685732344832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 666276259840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482ÌÆ'Ò±M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6618393 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36517497059328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2442593 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183773319168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183773319168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4176057 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 $ _timestamp‚1.784497299676768e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36707175679547 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7203726 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35573785511936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 666188231680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9612536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7203728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2442339 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 664189441661 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚75 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 _runtime‚ 373.58789639 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 666188231680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16826419 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4176054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36043072386464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16816262 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 685732344832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 666276259840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9622691 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16816262 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 685551937536 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180407296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16826419 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6618650 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34888053167104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36517480019968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 664103293083 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7203726 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 666276259840 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9622691 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9612536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712Käeüõ0Ó+ _wandbruntime‚373  _runtime‚373™¾sÿÔù 0loss/FRt2/assignment_loss‚2.9649279878997223 .loss/FBt1/detection_loss‚10.96858696029038 /loss/FBt2/detection_loss‚3.8835109669805274 trainer/global_step‚949 _step‚76 /loss/FRt1/detection_loss‚0.3667601386102066 $ _timestamp‚1.784497299677718e+09 1loss/SRqqt2/assignment_loss‚1.489669283389142 1loss/FBt1/assignment_loss‚0.25624018751630884 1loss/SRqqt1/detection_loss‚3.1481202287415253 1loss/SRqqt2/detection_loss‚1.0086680054753576 epoch‚1 _runtime‚ 373.58841562 0loss/FRt1/assignment_loss‚4.0145960035124375 2loss/SRqqt1/assignment_loss‚1.8801317444922012 0loss/FBt2/assignment_loss‚0.2071948056777315 /loss/FRt2/detection_loss‚1.5768653142163758 %loss/total_loss‚31.76527162680192'­:UÕý 0loss/FRt1/assignment_loss‚4.0145960035124375 2loss/SRqqt1/assignment_loss‚1.8801317444922012 0loss/FBt2/assignment_loss‚0.2071948056777315 /loss/FRt2/detection_loss‚1.5768653142163758 %loss/total_loss‚31.76527162680192 0loss/FRt2/assignment_loss‚2.9649279878997223 .loss/FBt1/detection_loss‚10.96858696029038 /loss/FBt2/detection_loss‚3.8835109669805274 trainer/global_step‚949 _step‚76 /loss/FRt1/detection_loss‚0.3667601386102066 $ _timestamp‚1.784497299677718e+09 1loss/SRqqt2/assignment_loss‚1.489669283389142 1loss/FBt1/assignment_loss‚0.25624018751630884 1loss/SRqqt1/detection_loss‚3.1481202287415253 1loss/SRqqt2/detection_loss‚1.0086680054753576 epoch‚1 _runtime‚ 373.58841562LȾbÿ(Ö‚(: •‰õÒÑŸ½+ disk.nvme0n1.in‚0$ disk.nvme0n1.out‚934.15283203125 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚934.15283203125& proc.memory.availableMB‚ 149.453125$ disk.nvme1c1n1.out‚ 260.556640625 disk.md0.in‚0 disk.md0.out‚ 14.7421875 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚132.4521484375 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1165.9186363220215 disk.nvme1c1n1.in‚0$ memory_percent‚99.08781051635742' proc.memory.percent‚17.218017578125 network.sent‚ 1.00112e+06) disk./.usagePercent‚75.94324288044433" disk.nvme1n1.out‚ 260.556640625" disk.dm-0.out‚1031.22802734375 proc.memory.rssMB‚2821 cpu‚26.060860744122706 network.recv‚148820Èý4Kš×‚(: •‰õÒÒÑ®, gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.996' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.99866666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù¿;xØ‚(: ¤‰õÒØáÌ+ disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1c1n1.out‚ 265.90625 disk.dm-1.in‚0 disk.dm-1.out‚135.8271484375 cpu‚26.056719004847565 network.recv‚153509 disk.rbd0.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 937.142578125 disk.md0.out‚ 16.58984375! disk.dm-0.out‚1032.4306640625 proc.memory.rssMB‚2821 proc.cpu.threads‚17 network.sent‚ 1.065447e+06$ disk./.usageGB‚1165.918556213379 disk.rbd0.out‚0 disk.md0.in‚0' proc.memory.percent‚17.218017578125) disk./.usagePercent‚75.94323766249008$ disk.nvme0c0n1.out‚ 937.142578125 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 265.90625 disk.dm-0.in‚0$ memory_percent‚99.09038543701172% proc.memory.availableMB‚ 149.03125îjŠY›Ù‚(: ¤‰õÒ¥Ñô+ gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚189.235' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.07833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ï1F0Ú+ _wandbruntime‚392  _runtime‚392@jJÿÃÛ½ _runtime‚ 392.175321204 _step‚77 & lr-AdamW/pg1‚0.0001017510384865864 & lr-AdamW/pg2‚0.0001017510384865864 trainer/global_step‚999 $ _timestamp‚1.784497318264396e+09‚`eÇÜÁ & lr-AdamW/pg1‚0.0001017510384865864 & lr-AdamW/pg2‚0.0001017510384865864 trainer/global_step‚999 $ _timestamp‚1.784497318264396e+09 _runtime‚ 392.175321204 _step‚77MvÙ/ƒ0Ý+ _wandbruntime‚392  _runtime‚392ÓÊ>¼®'Þ¨O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39047984938496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4386264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2563194 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7562455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38547060523751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39047984938496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 699447863808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39047879878144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37357090638848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4386267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39047879878144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚78 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38348449053696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 699535884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38547163711577 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 $ _timestamp‚1.784497318265241e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17671988 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10099390 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37849820746064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 719974766592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39047879878144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37357090638848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4386267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39047879878144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚78 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38348449053696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 699535884800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38547163711577 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 $ _timestamp‚1.784497318265241e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17671988 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10099390 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37849820746064 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 719974766592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 _step‚79 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 700214001664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38385815420928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38584608542955 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7569778 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 H@DeviceStatsMonilÛ?)òtor.on_train_batch_end/inactive_split.all.current‚242 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17689221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚999 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39085924355072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 720666191360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 720492075520 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7569776 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17679064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2565615 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37393477220864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844973185233262e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 698018917069 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7569776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39086029422592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17689221 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6956172 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37393278201344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36672786125824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37886692813824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10109288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6955930 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4390554 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38385815420928 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38385798381568 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 700214001664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4390557 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37886675774464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360}+‚w½&â·M M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 _step‚79 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 700214001664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38385815420928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38584608542955 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7569778 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17689221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚999 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39085924355072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 720666191360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 720492075520 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7569776 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17679064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2565615 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37393477220864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844973185233262e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 698018917069 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7569776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39086029422592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17689221 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6956172 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37393278201344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36672786125824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37886692813824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10109288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6955930 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4390554 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38385815420928 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38385798381568 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 700214001664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4390557 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37886675774464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 731172440921 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18524647 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39693407094704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7928507 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10596142 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40950135981568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4600767 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10606283 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40216750375936 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7287160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2686393 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 731086292455 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40950135981568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40216767415296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38421848830976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40424493387159 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7844973371292064e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10596142 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39693424134064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7928507 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 754962977792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7928505 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 754780466176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7928505 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40216767415296 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 733473626624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39176629297152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 I{l í®O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7928507 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 754962977792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7928505 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 754780466176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7928505 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40216767415296 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 733473626624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39176629297152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 731172440921 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18524647 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39693407094704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7928507 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10596142 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40950135981568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚219 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4600767 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10606283 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40216750375936 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7287160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2686393 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 731086292455 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40950135981568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40216767415296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38421848830976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40424493387159 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7844973371292064e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10596142 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39693424134064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352R’$?X0î+ _wandbruntime‚411  _runtime‚411ðÄňº&ï´M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40254116743168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40254116743168 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 734063715328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18541866 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7293555 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40254133782528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 734151743488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38457568891904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10616195 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38457543988224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚1049 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 734151743488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844973373877008e+09 _runtime‚ 411.298755426 _step‚83 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 734063715328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10606040 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4605054 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18552023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18541866 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2688501 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40462041406363 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18552023 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7935826 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 755660445184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7935828 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7293798 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40254133782528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2688741 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40988180458496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40462144594301 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40988285526016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 731848392477 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39213017734656 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7935826 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10606040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4605057 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39213229337088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39730279162464 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 731762243899 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40988285526016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 755473746432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0yÁ’¾&ð¸M [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7293555 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40254133782528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 734151743488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38457568891904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10616195 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38457543988224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚1049 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 734151743488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844973373877008e+09 _runtime‚ 411.298755426 _step‚83 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 734063715328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10606040 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4605054 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18552023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18541866 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2688501 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40462041406363 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18552023 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7935826 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 755660445184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7935828 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7293798 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40254133782528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2688741 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40988180458496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40462144594301 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40988285526016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 731848392477 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39213017734656 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7935826 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10606040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4605057 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39213229337088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39730279162464 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 731762243899 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40988285526016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 755473746432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40254116743168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40254116743168 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 734063715328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18541866 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360SWì.A0ñ+ _wandbruntime‚411  _runtime‚411™ð¶òû trainer/global_step‚1049 % _timestamp‚1.7844973373885953e+09 2loss/SRqqt1/assignment_loss‚1.6439178375394217 .loss/FRt2/detection_loss‚1.172972291552772 .loss/FBt2/detection_loss‚4.266559588369107 /loss/FRt2/assignment_loss‚2.785948846229287 /loss/FBt1/assignment_loss‚0.211718645420743 0loss/FRt1/detection_loss‚0.43358233613474545 1loss/SRqqt2/detection_loss‚0.9612140376200399 /loss/FBt1/detection_loss‚10.420996641153096 _runtime‚ 411.299226287 _step‚84 1loss/FBt2/assignment_loss‚0.05074112870460475 &loss/total_loss‚30.381159450839498 epoch‚1 /loss/FRt1/assignment_loss‚4.021933206397554 2loss/SRqqt2/assignment_loss‚1.2579662228589723 1loss/SRqqt1/detection_loss‚3.153608668859152379@ñóÿ epoch‚1 /loss/FRt1/assignment_loss‚4.021933206397554 2loss/SRqqt2/assignment_loss‚1.2579662228589723 1loss/SRqqt1/detection_loss‚3.1536086688591523 trainer/global_step‚1049 % _timestamp‚1.7844973373885953e+09 2loss/SRqqt1/assignment_loss‚1.6439178375394217 .loss/FRt2/detection_loss‚1.172972291552772 .loss/FBt2/detection_loss‚4.266559588369107 /loss/FRt2/assignment_loss‚2.785948846229287 /loss/FBt1/assignment_loss‚0.211718645420743 0loss/FRt1/detection_loss‚0.43358233613474545 1loss/SRqqt2/detection_loss‚0.9612140376200399 /loss/FBt1/detection_loss‚10.420996641153096 _runtime‚ 411.299226287 _step‚84 1loss/FBt2/assignment_loss‚0.05074112870460475 &loss/total_loss‚30.381159450839498TQðžö)ô‚(:ž ‰õÒ¬ÿÖ+ disk.md0.out‚ 18.23828125" disk.dm-0.out‚1061.92236328125 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚26.049316610260878) disk./.usagePercent‚75.88997005232187 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 270.564453125 disk.rbd0.out‚0$ memory_percent‚99.09141063690186( proc.memory.availableMB‚ 148.86328125 proc.memory.rssMB‚2821' proc.memory.percent‚17.218017578125 network.recv‚162195% disk./.usageGB‚1165.1007652282715 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 966.435546875 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 966.435546875 disk.md0.in‚0 disk.dm-0.in‚0! disk.dm-1.out‚136.84814453125 proc.cpu.threads‚17 network.sent‚ 1.145014e+06 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 270.564453125Õð0õ‚(:„ ‰õÒ—íý+ gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.466' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“QÎ0ö+ _wandbruntime‚429  _runtime‚429â†ÑVÇ÷Á ' lr-AdamW/pg1‚0.00010174295496359095 ' lr-AdamW/pg2‚0.00010174295496359095 trainer/global_step‚1099 % _timestamp‚1.7844973556871998e+09 _runtime‚ 429.600398823 _step‚85‰ñBËøÅ _runtime‚ 429.600398823 _step‚85 ' lr-AdamW/pg1‚0.00010174295496359095 ' lr-AdamW/pg2‚0.00010174295496359095 trainer/global_step‚1099 % _timestamp‚1.7844973556871998e+09UC0ú?0ù+ _wandbruntime‚429  _runtime‚429®ÿCÎúªO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11103035 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8294555 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19387449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42085068737536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8294555 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40996563663360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [HDevic‹Ë‚¨â eStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41537010482704 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7625265 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2809998 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180414464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42301926250567 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 767411368448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4815264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2809743 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40206631597056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42852497145344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42852392084992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42085068737536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 767411368448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 767323347456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 trainer/global_step‚1099 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205318144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 789751651840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11103035 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11092894 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 767323347456 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42852392084992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7625007 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 764915767863 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42085085776896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40206606693376 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 789932066304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0`ãÅ`´'û®O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11103035 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11092894 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 767323347456 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42852392084992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7625007 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 764915767863 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42085085776896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40206606693376 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 789932066304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11103035 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8294555 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19387449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42085068737536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8294555 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40996563663360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41537010482704 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7625265 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2809998 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180414464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42301926250567 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 767411368448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4815264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2809743 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40206631597056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42852497145344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42852392084992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42085068737536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 767411368448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 767323347456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 trainer/global_step‚1099 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205318144 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 789751651840Vó(0ü+ _wandbruntime‚429  _runtime‚429ÛÓ£=º&ý´M E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40242326754304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1099 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7631851 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42122452144128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7632103 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178310144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11112947 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42122435104768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42890541629440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 768089485312 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42890541629440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 790631000064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41573882550464 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19414825 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42339577457709 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 768001457152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11112947 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4819554 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8301876 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40242301850624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11102792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4819557 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19414825 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 765677867885 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7844973559452755e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42122435104768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19404668 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 429.856353508 _step‚87 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8301878 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42122452144128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2812546 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8301878 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203213824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41573899589824 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42890436561920 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8301876 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248Š Â]þ¸M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 429.856353508 _step‚87 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8301878 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42122452144128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2812546 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8301878 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203213824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41573899589824 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42890436561920 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8301876 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40242326754304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1099 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7631851 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42122452144128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7632103 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178310144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11112947 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42122435104768 F@DeviceStatsMonÊ4jÃaitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42890541629440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 768089485312 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42890541629440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 790631000064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41573882550464 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19414825 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42339577457709 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 768001457152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11112947 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4819554 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8301876 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40242301850624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11102792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4819557 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19414825 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 765677867885 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7844973559452755e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42122435104768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19404668 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0W\ “0ÿ+ _wandbruntime‚429  _runtime‚429„ú€û .loss/FBt2/detection_loss‚7.278968413784142 trainer/global_step‚1099 1loss/SRqqt2/assignment_loss‚1.398097419062938 1loss/FBt1/assignment_loss‚0.26162062308900536 0loss/FBt2/assignment_loss‚0.1690333368822134 epoch‚1 &loss/total_loss‚28.147356991034115 0loss/FRt2/assignment_loss‚2.8933593366253243 2loss/SRqqt1/assignment_loss‚1.6586373447337084 .loss/FBt1/detection_loss‚4.553513469619191 _step‚88 /loss/FRt2/detection_loss‚1.4223326662675677 % _timestamp‚1.7844973559461937e+09 _runtime‚ 429.856817909 /loss/FRt1/assignment_loss‚4.064472568198426 0loss/FRt1/detection_loss‚0.45456472706733025 0loss/SRqqt1/detection_loss‚3.124539670641101 1loss/SRqqt2/detection_loss‚0.8682174150631655@òù+ÿ 0loss/FRt1/detection_loss‚0.45456472706733025 0loss/SRqqt1/detection_loss‚3.124539670641101 1loss/SRqqt2/detection_loss‚0.8682174150631655 .loss/FBt2/detection_loss‚7.278968413784142 trainer/global_step‚1099 1loss/SRqqt2/assignment_loss‚1.398097419062938 1loss/FBt1/assignment_loss‚0.26162062308900536 0loss/FBt2/assignment_loss‚0.1690333368822134 epoch‚1 &loss/total_loss‚28.147356991034115 0loss/FRt2/assignment_loss‚2.8933593366253243 2loss/SRqqt1/assignment_loss‚1.6586373447337084 .loss/FBt1/detection_loss‚4.553513469619191 _step‚88 /loss/FRt2/detection_loss‚1.4223326662675677 % _timestamp‚1.7844973559461937e+09 _runtime‚ 429.856817909 /loss/FRt1/assignment_loss‚4.064472568198426XÅ–/‚‚(:¤ щõÒÕ´ì+ disk.md0.in‚0 disk.nvme0c0n1.in‚0 proc.memory.rssMB‚2821 proc.cpu.threads‚17% disk./.usageGB‚1165.1008834838867 disk.rbd0.in‚0( proc.memory.availableMB‚ 148.76953125 cpu‚26.044856020896333" disk.dm-0.out‚1069.05419921875 network.sent‚ 1.206318e+06 network.recv‚167257& disk.nvme1c1n1.out‚271.35107421875 disk.dm-0.in‚0& disk.nvme0c0n1.out‚973.89794921875 disk.nvme0n1.in‚0' proc.memory.percent‚17.218017578125 disk.nvme1n1.in‚0 disk.md0.out‚ 18.6328125 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚973.89794921875# memory_percent‚99.0919828414917! disk.dm-1.out‚137.10107421875) disk./.usagePercent‚75.88997775501623 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚271.35107421875%­Q™ƒ‚(:Ž щõÒ‚”, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.67' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.223333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0< ‘?0„+ _wandbruntime‚448  _runtime‚448A4XÓÇ…Á ' lr-AdamW/pg1‚0.00010173769112537171 ' lr-AdamW/pg2‚0.00010173769112537171 trainer/global_step‚1149 % _timestamp‚1.7844973743616574e+09 _runtime‚ 448.272585479 _step‚89pΈŠ_runtime‚ 448.272585479 _step‚89 ' lr-AdamW/pg1‚0.00010173769112537171 ' lr-AdamW/pg2‚0.00010173769112537171 trainer/global_step‚1149 % _timestamp‚1.7844973743616574e+09Y“„0‡+ _wandbruntime‚448  _runtime‚448¸>/°'ˆªO SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8660607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798831391737 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44754648188416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 824738416640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43953387099136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42816102972416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚90 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 824929316864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20250251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43953404138496 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 801349110272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1149 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41991364555776 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44754648188416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42816318776320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11599787 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44754753248768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7963701 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7963458 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43953387099136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2933934 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11589646 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 801261089280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44754753248768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 _runtime‚ 448.273285331 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11599787 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44179462301801 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5029764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43380613870704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43953404138496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 798745243271 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11589646 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8660607 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44179359113975 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8660605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43380630910064ã—àÙ´'‰®O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7963701 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7963458 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43953387099136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2933934 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11589646 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 801261089280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44754753248768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 _runtime‚ 448.273285331 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11599787 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44179462301801 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5029764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43380613870704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43953404138496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 798745243271 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11589646 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8660607 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44179359113975 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8660605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43380630910064 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8660607 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798831391737 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44754648188416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 824738416640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43953387099136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42816102972416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚90 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 824929316864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20250251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43953404138496 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 801349110272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1149 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41991364555776 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44754648188416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42816318776320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11599787 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44754753248768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992Z‰cÜn0Š+ _wandbruntime‚448  _runtime‚448BAø¹ ‹´M FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7970177 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42852705633792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 825438512640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7970429 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44792797732864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 801939198976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43417485938464 _runtime‚ 448.524840827 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11599544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44217010321117 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43417502977824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42027084616704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8667926 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8667926 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5034057 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2936123 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42027059713024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799507343293 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20277627 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11599544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20277627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 trainer/global_step‚1149 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 802027227136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 802027227136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11609699 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44216907133179 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20267470 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20267470 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43990753466368 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 799421194715 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8667928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5034054 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 % _timestamp‚1.7844973746137667e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 825621017088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44792797732864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2936372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43990770505728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43990753466368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8667928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42852498225664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 197829632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20277627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 trainer/global_step‚1149 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 802027227136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 802027227136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11609699 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44216907133179 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20267470 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20267470 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43990753466368 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 799421194715 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8667928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5034054 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 % _timestamp‚1.7844973746137667e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 825621017088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44792797732864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2936372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43990770505728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43990753466368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8667928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42852498225664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 197829632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7970177 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42852705633792 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 825438512640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7970429 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44792797732864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 801939198976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43417485938464 _runtime‚ 448.524840827 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11599544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44217010321117 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43417502977824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42027084616704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8667926 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8667926 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5034057 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2936123 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42027059713024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799507343293 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20277627 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11599544[óU«0+ _wandbruntime‚448  _runtime‚448ŽÈ›´ýŽ÷ 1loss/FBt2/assignment_loss‚0.10246258354478364 /loss/FRt1/detection_loss‚0.3908567354599877 0loss/SRqqt2/detection_loss‚1.029396993174039 .loss/FBt1/detection_loss‚5.462687994093123 epoch‚1 /loss/FRt1/assignment_loss‚3.948119532099009 %loss/total_loss‚29.96448640260811 _runtime‚ 448.525362218 1loss/SRqqt2/assignment_loss‚1.232920343869491 /loss/FRt2/detection_loss‚1.3084239385309164 /loss/SRqqt1/detection_loss‚3.25118587243882 .loss/FBt2/detection_loss‚8.537643322821808 trainer/global_step‚1149 % _timestamp‚1.7844973746146417e+09 _step‚92 0loss/FRt2/assignment_loss‚2.7823653770585586 1loss/SRqqt1/assignment_loss‚1.755369943069915 1loss/FBt1/assignment_loss‚0.16305376644765815ãmû /loss/FRt1/detection_loss‚0.3908567354599877 0loss/SRqqt2/detection_loss‚1.029396993174039 .loss/FBt1/detection_loss‚5.462687994093123 epoch‚1 /loss/FRt1/assignment_loss‚3.948119532099009 %loss/total_loss‚29.96448640260811 _runtime‚ 448.525362218 1loss/SRqqt2/assignment_loss‚1.232920343869491 /loss/FRt2/detection_loss‚1.3084239385309164 /loss/SRqqt1/detection_loss‚3.25118587243882 .loss/FBt2/detection_loss‚8.537643322821808 trainer/global_step‚1149 % _timestamp‚1.7844973746146417e+09 _step‚92 0loss/FRt2/assignment_loss‚2.7823653770585586 1loss/SRqqt1/assignment_loss‚1.755369943069915 1loss/FBt1/assignment_loss‚0.16305376644765815 1loss/FBt2/assignment_loss‚0.10246258354478364\]bG(0‚(:¥ à‰õÒ÷†¶+ network.sent‚ 1.256045e+06) disk./.usagePercent‚75.87450602372566 disk.md0.in‚0 disk.dm-0.in‚0& disk.nvme1c1n1.out‚272.04541015625$ memory_percent‚99.09412860870361 cpu‚26.042913366426564 network.recv‚171929 disk.md0.out‚ 19.19140625" disk.dm-0.out‚1069.57763671875 disk.dm-1.in‚0! disk.dm-1.out‚137.14013671875& disk.nvme0c0n1.out‚974.95166015625 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚974.95166015625$ disk./.usageGB‚1164.863353729248 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚272.04541015625( proc.memory.availableMB‚ 148.41796875 proc.memory.rssMB‚2821' proc.memory.percent‚17.218017578125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17¥GŠÿš‘‚(: à‰õÒÐú+ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚191.828' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.94266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðWÀÚ™’‚(:Ž ï‰õÒÛÛµ+ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚197.48' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.82666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûðr)“‚(:ž ï‰õÒ¨¬Ë+ network.sent‚ 1.320218e+06 disk.nvme0n1.in‚0" disk.nvme1n1.out‚ 273.736328125( proc.memory.availableMB‚ 148.36328125' proc.memory.percent‚17.218017578125( disk./.usagePercent‚75.8917446537139" disk.dm-0.out‚1077.16748046875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 273.736328125 disk.nvme1n1.in‚0$ memory_percent‚99.09446239471436 proc.cpu.threads‚17% disk./.usageGB‚1165.1280097961426 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 137.669921875 disk.nvme0c0n1.in‚0 network.recv‚176855 disk.md0.out‚ 19.55859375 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚982.17138671875$ disk.nvme0n1.out‚982.17138671875 proc.memory.rssMB‚2821 cpu‚26.04311663546512¶‰•0”+ _wandbruntime‚466  _runtime‚466n›GÌÅ•¿ & lr-AdamW/pg1‚0.0001017316126774032 & lr-AdamW/pg2‚0.0001017316126774032 trainer/global_step‚1199 % _timestamp‚1.7844973923757868e+09 _runtime‚ 466.286717771 _step‚93 Nv0É–Ã & lr-AdamW/pg2‚0.0001017316126774032 trainer/global_step‚1199 % _timestamp‚1.7844973923757868e+09 _runtime‚ 466.286717771 _step‚93 & lr-AdamW/pg1‚0.0001017316126774032]’.0—+ _wandbruntime‚466  _runtime‚466€‹Wi°'˜ªO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832660867145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45821705460736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1199 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9026657 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9026655 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 835198831104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5244264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12086398 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43776122418176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45224217258704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46657009352192 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8301979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46656904291840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12096539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832574718679 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46056791977383 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844973923765936e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835286852096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44635782091264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21123196 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9026655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3057474 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 859846379008 _runtime‚ 466.290502887 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8301738 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21113053 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12086398 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705920 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46056895165209 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45224234298064 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46656904291840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835286852096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12096539 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46056791977383 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844973923765936e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatscŸ_4"Monitor.on_train_batch_start/segment.small_pool.current‚152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835286852096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44635782091264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21123196 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9026655 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3057474 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 859846379008 _runtime‚ 466.290502887 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8301738 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21113053 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12086398 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705920 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46056895165209 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45224234298064 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46656904291840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835286852096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12096539 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832660867145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45821705460736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1199 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9026657 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9026655 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 835198831104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5244264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12086398 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43776122418176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45224217258704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46657009352192 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8301979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46656904291840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12096539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832574718679 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12096296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21140429 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12106451 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46094339996587 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45859071827968 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9033978 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8308653 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚95 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9033978 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46695053836288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211602432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 833250670123 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 833336818701 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43811817575424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46694948768768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8308402 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 860545665024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21130272 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9033976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45859071827968 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44672388144128 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44672176541696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21140429 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 _runtime‚ 466.539281198 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45859088867328 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45859088867328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43811842479104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46094443184525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45261106365824 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45859088867328 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45859088867328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43811842479104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46094443184525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45261106365824 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12096296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21140429 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12106451 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46094339996587 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45859071827968 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9033978 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8308653 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚95 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9033978 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46695053836288 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211602432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 833250670123 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 833336818701 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43811817575424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46694948768768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8308402 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 860545665024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21130272 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9033976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45859071827968 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44672388144128 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44672176541696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21140429 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 _runtime‚ 466.539281198 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5458767 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMoq¹A~Ï$nitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47934328028617 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 869224593920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9392705 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47690040861696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12593291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48559265455616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47690023822336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48559160395264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47934224840791 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8638802 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12583150 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21985998 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47067837686064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9392707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45560905184256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48559160395264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 866404194087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 869224593920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12583150 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5458764 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8638802 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12583150 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21985998 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47067837686064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9392707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45560905184256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48559160395264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 866404194087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 869224593920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12583150 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5458764 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5458767 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47934328028617 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 869224593920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9392705 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47690040861696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12593291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48559265455616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47690023822336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48559160395264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0b ™h0¨+ _wandbruntime‚484  _runtime‚484ZÎÀº&©´M YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47727390189568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8645488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 869814682624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45596575437824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21993074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48597309939712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46492085787136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3182666 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9400026 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22003231 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12603203 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 869902710784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46491876281856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12593048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 197829632 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47971876047933 _step‚99 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5463057 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12603203 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22003231 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8645723 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3182434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209505280 % _timestamp‚1.7844974106696935e+09§¼r ª¸M D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9400026 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22003231 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12603203 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 869902710784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46491876281856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12593048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 197829632 VCDeviceStatsR¯a LMonitor.on_train_batch_end/requested_bytes.all.allocated‚47971876047933 _step‚99 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5463057 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12603203 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22003231 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8645723 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3182434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209505280 % _timestamp‚1.7844974106696935e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47727390189568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8645488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 869814682624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45596575437824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21993074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48597309939712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46492085787136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3182666 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48911424034704 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3302720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48911441074064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50461416498688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5673264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49558342183936 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 903162335744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22848800 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13079902 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 903074314752 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 903162335744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48275347422720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201123840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47345663046656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7844974282041843e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 903074314752 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176220160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49811760892025 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 900233669495 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9758755 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22838657 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50461521559040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49558359223296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49558342183936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8975984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3302949 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 900319817961 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13079902Ù’<̵'µ¯O VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 903074314752 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 903162335744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48275347422720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201123840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47345663046656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7844974282041843e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 903074314752 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176220160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49811760892025 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 900233669495 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9758755 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22838657 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50461521559040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49558359223296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49558342183936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8975984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3302949 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 900319817961 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13079902 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚49558359223296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48911424034704 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3302720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48911441074064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50461416498688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5673264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49558342183936 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 903162335744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22848800 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13079902 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330fWÑT§0¶+ _wandbruntime‚502  _runtime‚502•Aé•Ð·´M 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48948296102464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22866033 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13089800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8982942 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50499566043136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49595725590528 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9766078 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49849308911341 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBD7¶ÓêeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 903840452608 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 903752424448 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 930387631104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5677554 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49595708551168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13089800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22855876 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49595708551168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48311538426880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182504448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9766078 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 903752424448 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50499566043136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3305385 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47381358203904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5677557 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50499460975616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13099955 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22866033 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8982692 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48311745835008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13099955 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47381333300224 _step‚103 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904@¸p¾&¸¸M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49595708551168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48311538426880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182504448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9766078 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 903752424448 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50499566043136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3305385 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47381358203904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5677557 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50499460975616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13099955 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22866033 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8982692 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48311745835008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13099955 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47381333300224 _step‚103 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48948296102464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22866033 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13089800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8982942 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50499566043136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49595725590528 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9766078 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49849308911341 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 903840452608 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 903752424448 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 930387631104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5677554 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49595708551168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13089800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22855876 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104g‡°`Ÿ0¹+ _wandbruntime‚502  _runtime‚502 ©Rºû trainer/global_step‚1299 .loss/FRt1/assignment_loss‚3.96795592819833 0loss/FRt2/assignment_loss‚2.4172738893220655 1loss/FBt1/assignment_loss‚0.24410065929930624 1loss/SRqqt2/detection_loss‚0.8365119733522749 _runtime‚ 502.367756851 _step‚104 /loss/FRt1/detection_loss‚0.5223321495746068 0loss/SRqqt1/detection_loss‚2.683667182602596 .loss/FBt1/detection_loss‚6.758031897273175 .loss/FBt2/detection_loss‚2.908752892794048 % _timestamp‚1.7844974284570346e+09 1loss/SRqqt1/assignment_loss‚1.519138478885303 1loss/FBt2/assignment_loss‚0.06915964806946497 &loss/total_loss‚24.457575450405145 2loss/SRqqt2/assignment_loss‚1.2232208662624384 /loss/FRt2/detection_loss‚1.3074298847715322 epoch‚1̦N»ÿ 2loss/SRqqt2/assignment_loss‚1.2232208662624384 /loss/FRt2/detection_loss‚1.3074298847715322 epoch‚1 trainer/global_step‚1299 .loss/FRt1/assignment_loss‚3.96795592819833 0loss/FRt2/assignment_loss‚2.4172738893220655 1loss/FBt1/assignment_loss‚0.24410065929930624 1loss/SRqqt2/detection_loss‚0.8365119733522749 _runtime‚ 502.367756851 _step‚104 /loss/FRt1/detection_loss‚0.5223321495746068 0loss/SRqqt1/detection_loss‚2.683667182602596 .loss/FBt1/detection_loss‚6.758031897273175 .loss/FBt2/detection_loss‚2.908752892794048 % _timestamp‚1.7844974284570346e+09 1loss/SRqqt1/assignment_loss‚1.519138478885303 1loss/FBt2/assignment_loss‚0.06915964806946497 &loss/total_loss‚24.457575450405145he‰Ø¹0¼‚(:¥ œŠõÒâèÐ+ disk.nvme0n1.in‚0% disk.nvme0n1.out‚1956.64990234375" disk.dm-0.out‚2074.14111328125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚299.06201171875 disk.md0.out‚ 20.6015625 proc.cpu.threads‚17% disk./.usageGB‚1166.1280708312988 disk.nvme1n1.in‚0' proc.memory.percent‚17.218017578125 network.recv‚189892) disk./.usagePercent‚75.95688460063825 disk.dm-0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚138.22509765625 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚299.06201171875 disk.md0.in‚0 network.sent‚ 1.461125e+06 disk.rbd0.in‚0' disk.nvme0c0n1.out‚1956.64990234375$ memory_percent‚99.88284111022949& proc.memory.availableMB‚ 19.1953125 proc.memory.rssMB‚2821 cpu‚26.050395149372555RÛŽŽ½‚(:ƒ œŠõÒö¥ú+ gpu.0.gpu‚77.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.89' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.63 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.õ<80¾+ _wandbruntime‚520  _runtime‚520úÄk0È¿Â ' lr-AdamW/pg1‚0.00010170849071381748 ' lr-AdamW/pg2‚0.00010170849071381748 trainer/global_step‚1349 % _timestamp‚1.7844974467367861e+09 _runtime‚ 520.647704778 _step‚105(³±ÃÌÀÆ _step‚105 ' lr-AdamW/pg1‚0.00010170849071381748 ' lr-AdamW/pg2‚0.00010170849071381748 trainer/global_step‚1349 % _timestamp‚1.7844974467367861e+09 _runtime‚ 520.647704778i•Ëðç0Á+ _wandbruntime‚520  _runtime‚520³ýµ'¯O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13576654 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937012056576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 964411604992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9313749 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50755027422704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 937100077568 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49130420909056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3425982 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1349 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52363672602112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23701459 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23711602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50755044462064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13586795 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51426660545536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10124807 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51426660545536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13586795 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10124805 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 937100077568 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9313481 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 934063144903 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52363777662464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52363777662464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844974467376018e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10124807 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13576654 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 _runtime‚ 520.648387359 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5887767 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50094807610368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10124805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330’³—Ç`óO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51426660545536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10124807 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51426660545536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13586795 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10124805 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 937100077568 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9313481 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 934063144903 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52363777662464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52363777662464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allÚ‹-Y!ocated_bytes.all.current‚ 105060352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844974467376018e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10124807 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13576654 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 _runtime‚ 520.648387359 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5887767 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50094807610368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10124805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13576654 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937012056576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 964411604992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9313749 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50755027422704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 937100077568 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49130420909056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3425982 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1349 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52363672602112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23701459 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23711602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50755044462064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13586795 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2j%ëð0Ä+ _wandbruntime‚520  _runtime‚520—/Tø¿&ŹM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1349 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49166116066304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23728835 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9320421 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50131410259456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 965113785856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50791916529824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52401822146560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13586552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13586552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10132126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚107 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10132128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13596707 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 937690166272 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 934739096347 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23718678 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3428364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51464043952128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52401717079040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180407296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23718678 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49166091162624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51726638586811 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 934825244925 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13596707 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 965294193152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50791899490464 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 937778194432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51726741774749 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚ 520.906271881 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 937778194432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9320170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52401822146560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23728835 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13596707 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 965294193152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50791899490464 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 937778194432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51726741774749 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚ 520.906271881 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 937778194432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9320170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52401822146560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23728835 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23728835 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9320421 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50131410259456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 965113785856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50791916529824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52401822146560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13586552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13586552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10132126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚107 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10132128 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13596707 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 937690166272 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 934739096347 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23718678 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3428364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51464043952128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52401717079040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180407296kŠö—K0Ç+ _wandbruntime‚520  _runtime‚520sÞ¶¹Èû 1loss/SRqqt2/assignment_loss‚1.148396036854798 /loss/FRt2/detection_loss‚1.3925972367095831 epoch‚1 % _timestamp‚1.7844974469961169e+09 /loss/FRt1/assignment_loss‚4.025792955423331 2loss/SRqqt1/assignment_loss‚1.5550844913969437 1loss/FBt1/assignment_loss‚0.23599564369427886 1loss/FBt2/assignment_loss‚0.05884342081353188 _runtime‚ 520.906732981 _step‚108 /loss/FRt1/detection_loss‚0.4875348994042724 1loss/SRqqt1/detection_loss‚2.8109334540535866 1loss/SRqqt2/detection_loss‚0.7728398240256196 -loss/FBt1/detection_loss‚4.72779902077577 %loss/total_loss‚23.46984606738385 /loss/FRt2/assignment_loss‚2.548398667160486 /loss/FBt2/detection_loss‚3.7056304170716503 trainer/global_step‚1349tbˆÉÿ %loss/total_loss‚23.46984606738385 /loss/FRt2/assignment_loss‚2.548398667160486 /loss/FBt2/detection_loss‚3.7056304170716503 trainer/global_step‚1349 1loss/SRqqt2/assignment_loss‚1.148396036854798 /loss/FRt2/detection_loss‚1.3925972367095831 epoch‚1 % _timestamp‚1.7844974469961169e+09 /loss/FRt1/assignment_loss‚4.025792955423331 2loss/SRqqt1/assignment_loss‚1.5550844913969437 1loss/FBt1/assignment_loss‚0.23599564369427886 1loss/FBt2/assignment_loss‚0.05884342081353188 _runtime‚ 520.906732981 _step‚108 /loss/FRt1/detection_loss‚0.4875348994042724 1loss/SRqqt1/detection_loss‚2.8109334540535866 1loss/SRqqt2/detection_loss‚0.7728398240256196 -loss/FBt1/detection_loss‚4.72779902077577l€‚oÑ1Ê‚(:¦ «ŠõÒÛÆÐ+ disk.rbd0.in‚0% disk.nvme0n1.out‚2188.68115234375 disk.nvme1c1n1.in‚0 network.sent‚ 1.523504e+06 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚322.80419921875' proc.memory.percent‚17.218017578125 cpu‚26.048525957308726) disk./.usagePercent‚75.95262053812235 disk.md0.out‚ 21.13671875 disk.dm-0.in‚0" disk.dm-0.out‚2328.44677734375 disk.nvme0n1.in‚0& proc.memory.availableMB‚ 19.1484375% disk./.usageGB‚1166.0626068115234 disk.md0.in‚0& disk.nvme1c1n1.out‚322.80419921875$ memory_percent‚99.88312721252441 proc.memory.rssMB‚2821 proc.cpu.threads‚17 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2188.68115234375 network.recv‚195036 disk.dm-1.in‚0! disk.dm-1.out‚138.53857421875áëI.™Ë‚(:Ž «ŠõÒ¸”÷+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.109' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.03633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ“ÕÖ0Ì+ _wandbruntime‚539  _runtime‚539à°CÈÍ _runtime‚ 539.530242596 _step‚109 ' lr-AdamW/pg1‚0.00010169915493089033 ' lr-AdamW/pg2‚0.00010169915493089033 trainer/global_step‚1399 % _timestamp‚1.7844974656192818e+09 '3€ÌÎÆ _step‚109 ' lr-AdamW/pg1‚0.00010169915493089033 ' lr-AdamW/pg2‚0.00010169915493089033 trainer/global_step‚1399 % _timestamp‚1.7844974656192818e+09 _runtime‚ 539.530242596m^Ž^m0Ï+ _wandbruntime‚539  _runtime‚539€pŸÐ¯O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54265928705536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10490855 % _timestamp‚1.7844974656201642e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9650051 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚1399 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6102267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚ 539.534417754 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3547787 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 999564779008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192997376 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53566523431015 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 967978768777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24574404 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54266033765888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14083547 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10490857 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 970949798400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10490855 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54266033765888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10490857 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53294995946496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253šZò‹¹'ѳO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24574404 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54266033765888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14083547 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10490857 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 970949798400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10490855 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54266033765888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10490857 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53294995946496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54265928705536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10490855 % _timestamp‚1.7844974656201642e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9650051 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚1399 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6102267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚ 539.534417754 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3547787 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 999564779008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192997376 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53566523431015 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 967978768777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104nQP½¦0Ò+ _wandbruntime‚539  _runtime‚539t±¯UÀ&ÓºM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1000075976192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24591637 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10498176 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51950925001216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 539.789317822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14083304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 971627908096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14093459 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14093459 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 971627908096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53332345274368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6106554 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178310144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54303973182464 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 971715936256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6106557 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3550254 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚111 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53332362313728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53604174638157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9656808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 $ _timestamp‚1.784497465878208e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53332345274368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53332362313728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1000254286336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54304078249984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24581480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53604071450219 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10498178 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52635519917824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 971715936256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50950849025024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10498176 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24591637 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 968568571755 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54304078249984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9657047 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3550490 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24581480yÓî¥ Ô¾M C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24591637 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 968568571755 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54304078249984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9657047 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3550490 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24581480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1000075976192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24591637 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10498176 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51950925001216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 539.789317822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14083304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1399 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 971627908096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14093459 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14093459 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.smalú¹ÑΦl_pool.freed‚ 971627908096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53332345274368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6106554 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178310144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54303973182464 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 971715936256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6106557 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3550254 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚111 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53332362313728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53604174638157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9656808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 $ _timestamp‚1.784497465878208e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53332345274368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53332362313728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1000254286336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54304078249984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24581480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53604071450219 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10498178 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52635519917824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 971715936256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50950849025024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10498176 O0ß+ _wandbruntime‚558  _runtime‚558Ë€¨½'à·O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1001722095719 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25427063 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53734243985408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56168184808960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1034514766848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14580299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10856907 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9987496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14570158 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9987254 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25437206 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3670490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1034332255232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 % _timestamp‚1.7844974845454895e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56168289869312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52699936633856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54442251238064 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54442234198704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55163297268736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56168289869312 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10856905 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6316767 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10856905 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1001808244185q]^ÌÁ'á»O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25437206 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3670490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1034332255232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 % _timestamp‚1.7844974845454895e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56168289869312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52699936633856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54442251238064 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54442234198704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55163297268736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56168289869312 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10856905 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6316767 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10856905 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1001808244185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1001722095719 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25427063 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53734243985408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56168184808960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1034514766848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14580299 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10856907 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9987496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14570158 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9987254ró!O·0â+ _wandbruntime‚558  _runtime‚558|*‡º“ã¿M D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52735631791104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182504448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10864226 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allo°dV2"cated‚10864228 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1002484195741 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25444282 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9993946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3673129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55200663635968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52735606887424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14590211 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6321054 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54479106266464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6321057 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10864228 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1035032637952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55200663635968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1005653678080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844974848021488e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1005565649920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1005565649920 _step‚115 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55200680675328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14590211 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1005653678080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25444282 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55481607501565 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1002398047163 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25454439 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10864226 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3672892 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53770846933504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56206334353408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1035215142400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56206229285888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 558.7133223 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55200680675328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14580056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25454439 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53770639525376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14580056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56206334353408 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56206229285888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 558.7133223 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55200680675328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14580056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25454439 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53770639525376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14580056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56206334353408 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52735631791104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182504448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10864226 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10864228 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1002484195741 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25444282 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9993946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3673129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55200663635968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52735606887424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14590211 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6321054 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54479106266464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6321057 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10864228 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1035032637952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55200663635968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1005653678080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844974848021488e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1005565649920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1005565649920 _step‚115 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55200680675328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14590211 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1005653678080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25444282 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55481607501565 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1002398047163 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25454439 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10864226 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3672892 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53770846933504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584s‰Qr0å+ _wandbruntime‚558  _runtime‚558æÇæû 0loss/FRt1/assignment_loss‚3.9423983331492765 2loss/SRqqt1/assignment_loss‚1.4434893936596134 1loss/FBt1/assignment_loss‚0.21683481840171023 /loss/FRt1/detection_loss‚0.5749572833436899 /loss/FBt1/detection_loss‚2.6385536941347527 .loss/FBt2/detection_loss‚4.138026986659199 _step‚116 1loss/SRqqt2/detection_loss‚0.8884922435803304 _runtime‚ 558.713809441 1loss/SRqqt2/assignment_loss‚1.117583404418664 0loss/FBt2/assignment_loss‚0.1601584620960542 trainer/global_step‚1449 /loss/FRt2/assignment_loss‚2.442042697495767 .loss/FRt2/detection_loss‚1.480465967831776 1loss/SRqqt1/detection_loss‚2.5379649224672964 %loss/total_loss‚21.58096820723813 epoch‚1 % _timestamp‚1.7844974848031588e+09sy|çÿ 1loss/SRqqt2/assignment_loss‚1.117583404418664 0loss/FBt2/assignment_loss‚0.1601584620960542 trainer/global_step‚1449 /loss/FRt2/assignment_loss‚2.442042697495767 .loss/FRt2/detection_loss‚1.480465967831776 1loss/SRqqt1/detection_loss‚2.5379649224672964 %loss/total_loss‚21.58096820723813 epoch‚1 % _timestamp‚1.7844974848031588e+09 0loss/FRt1/assignment_loss‚3.9423983331492765 2loss/SRqqt1/assignment_loss‚1.4434893936596134 1loss/FBt1/assignment_loss‚0.21683481840171023 /loss/FRt1/detection_loss‚0.5749572833436899 /loss/FBt1/detection_loss‚2.6385536941347527 .loss/FBt2/detection_loss‚4.138026986659199 _step‚116 1loss/SRqqt2/detection_loss‚0.8884922435803304 _runtime‚ 558.713809441te‘Ï"è‚(:— ØŠõÒ€çÜ+! disk.dm-0.out‚2346.4658203125 disk.dm-1.out‚ 138.873046875% disk.nvme0c0n1.out‚2207.060546875$ memory_percent‚99.88422393798828 cpu‚26.045200802454797 network.sent‚ 1.653621e+06) disk./.usagePercent‚75.95255121387308 disk.nvme1n1.in‚0# disk.nvme1n1.out‚325.0615234375 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme1c1n1.out‚325.0615234375 disk.md0.in‚0 disk.md0.out‚22.1875 disk.rbd0.out‚0 proc.memory.rssMB‚2821' proc.memory.percent‚17.218017578125 proc.cpu.threads‚17 network.recv‚207842 disk.nvme1c1n1.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚2207.060546875$ proc.memory.availableMB‚18.96875% disk./.usageGB‚1166.0615425109863T8'Üé‚(:„ ØŠõÒ´ˆ, gpu.0.gpu‚61.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚197.373' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.791 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é½a0ê+ _wandbruntime‚577  _runtime‚577ýá ˆÇëÁ ' lr-AdamW/pg1‚0.00010167804155006368 ' lr-AdamW/pg2‚0.00010167804155006368 trainer/global_step‚1499 $ _timestamp‚1.784497503448352e+09 _runtime‚ 577.359296535 _step‚117ZºZ>ËìÅ trainer/global_step‚1499 $ _timestamp‚1.784497503448352e+09 _runtime‚ 577.359296535 _step‚117 ' lr-AdamW/pg1‚0.00010167804155006368 ' lr-AdamW/pg2‚0.00010167804155006368urߣ¤0í+ _wandbruntime‚577  _runtime‚577 AD¿'î¹O LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1038913303040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 trainer/global_step‚1499 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10324757 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚118 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11222957 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58070545972736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10324997 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55554161622528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57321492345657 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57031615630336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15066910 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11222957 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55553954207232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1035637719593 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58070440912384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11222955 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1069467126272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844975034493186e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3793730 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56285837586704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1038913303040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1035551571127 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56285854626064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57031615630336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1038825282048 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58070545972736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 _runtime‚ 577.360290237 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58070440912384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1038825282048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26300008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1069284614656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6531267 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57031632669696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57321389157831 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57031632669696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 h+ï½O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26289865 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15077051 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54484669592576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 PCDeviceStatsMonit/+×s±%or.on_train_batch_start/active.small_pool.allocated‚15077051 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15066910 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6531264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1038825282048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26300008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1069284614656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6531267 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57031632669696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57321389157831 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57031632669696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1038913303040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 trainer/global_step‚1499 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10324757 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚118 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11222957 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58070545972736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10324997 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55554161622528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57321492345657 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57031615630336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15066910 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11222957 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55553954207232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1035637719593 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58070440912384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11222955 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1069467126272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844975034493186e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3793730 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56285837586704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1038913303040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1035551571127 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56285854626064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57031615630336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1038825282048 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58070545972736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 _runtime‚ 577.360290237 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58070440912384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 577.61886782 _step‚119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26307084 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10331459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3795905 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57359040364973 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56322726693824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11230276 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11230278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57068981997568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15076808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10331710 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56322709654464 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57068981997568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15086963 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184601600 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58108590456832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚1499 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26317241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55590552596480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54520389653504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1069978341376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26317241 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55590343091200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1036227522571 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6535557 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3796153 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1039503391744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15086963 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7844975037077868e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15076808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1036313671149ßsøÄÌ&òÆM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1039503391744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15086963 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7844975037077868e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15076808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1036313671149 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6535554 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54520364749824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209505280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 577.61886782 _step‚119 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26307084 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10331459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3795905 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57359040364973 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56322726693824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11230276 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11230278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57068981997568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15076808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10331710 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56322709654464 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57068981997568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15086963 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184601600 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58108590456832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚1499 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26317241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55590552596480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54520389653504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1069978341376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26317241 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55590343091200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1036227522571 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6535557 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3796153 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0wÒËÔ0ó+ _wandbruntime‚577  _runtime‚577Ñ@u^ôÿ 2loss/SRqqt2/assignment_loss‚1.3028948979947386 1loss/SRqqt2/detection_loss‚0.7808835649025966 /loss/FBt2/detection_loss‚2.9755732276269184 1loss/FBt2/assignment_loss‚0.08463740826763179 /loss/FRt1/detection_loss‚0.5135271424164867 trainer/global_step‚1499 % _timestamp‚1.7844975037087119e+09 0loss/FRt2/assignment_loss‚2.3927119447473895 1loss/FBt1/assignment_loss‚0.19377326953953672 /loss/FRt2/detection_loss‚1.3555583444897805 .loss/FBt1/detection_loss‚2.938422156783181 _step‚120 /loss/FRt1/assignment_loss‚3.852717215392633 2loss/SRqqt1/assignment_loss‚1.5378728436289337 1loss/SRqqt1/detection_loss‚2.5839600042772344 &loss/total_loss‚20.512532020067063 epoch‚1 _runtime‚ 577.619312571Übv õƒ 1loss/FBt2/assignment_loss‚0.08463740826763179 /loss/FRt1/detection_loss‚0.5135271424164867 trainer/global_step‚1499 % _timestamp‚1.7844975037087119e+09 0loss/FRt2/assignment_loss‚2.3927119447473895 1loss/FBt1/assignment_loss‚0.19377326953953672 /loss/FRt2/detection_loss‚1.3555583444897805 .loss/FBt1/detection_loss‚2.938422156783181 _step‚120 /loss/FRt1/assignment_loss‚3.852717215392633 2loss/SRqqt1/assignment_loss‚1.5378728436289337 1loss/SRqqt1/detection_loss‚2.5839600042772344 &loss/total_loss‚20.512532020067063 epoch‚1 _runtime‚ 577.619312571 2loss/SRqqt2/assignment_loss‚1.3028948979947386 1loss/SRqqt2/detection_loss‚0.7808835649025966 /loss/FBt2/detection_loss‚2.9755732276269184x| þØ,ö‚(:¡ çŠõÒÊç²+ network.sent‚ 1.702284e+06) disk./.usagePercent‚75.95255668030133 disk.dm-1.out‚ 139.029296875 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 19.1484375 network.recv‚212059" disk.dm-0.out‚2347.84033203125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1c1n1.out‚326.0478515625# disk.nvme1n1.out‚326.0478515625$ memory_percent‚99.88312721252441 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2208.94482421875 disk.nvme1n1.in‚0 disk.md0.in‚0 proc.memory.rssMB‚2821 proc.cpu.threads‚17% disk./.usageGB‚1166.0616264343262' disk.nvme0c0n1.out‚2208.94482421875 disk.md0.out‚ 22.8203125' proc.memory.percent‚17.218017578125 cpu‚26.044515142523096u›÷‚(: çŠõÒô¶Š, gpu.0.gpu‚76.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚171.527' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚57.17566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãW«0ø+ _wandbruntime‚595  _runtime‚595ðÄ('Èù _step‚121 ' lr-AdamW/pg1‚0.00010166626429033777 ' lr-AdamW/pg2‚0.00010166626429033777 trainer/global_step‚1549 % _timestamp‚1.7844975219841762e+09 _runtime‚ 595.895128568^1£™:úÆ ' lr-AdamW/pg2‚0.00010166626429033777 trainer"_†m’/global_step‚1549 % _timestamp‚1.7844975219841762e+09 _runtime‚ 595.895128568 _step‚121 ' lr-AdamW/pg1‚0.00010166626429033777yLJ 40û+ _wandbruntime‚595  _runtime‚595‹vt>¾'ü¸O @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58899951031296 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3916996 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58899951031296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1072851044864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58899933991936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10662515 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11589007 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57373885928448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15563662 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844975219851232e+09 _runtime‚ 595.89593744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1069381046535 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27152667 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59972802076160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1072763023872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15573803 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11589005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15563662 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59198822021239 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6745764 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58899933991936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180414464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59972697015808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6745767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56269427454976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58129440974704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1069467195001 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚1549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1072851044864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10662763 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11589005 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59972697015808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57373680610304 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15573803 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1104253155328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59972802076160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11589007 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27162810 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3916751 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚09ž!îÂ'ý¼O SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11589007 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27162810 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3916751 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58899951031296 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3916996 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58899951031296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1072851044864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58899933991936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10662515 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11589007 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57373885928448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15563662 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844975219851232e+09 _runtime‚ 595.89593744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1069381046535 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27152667 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59972802076160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1072763023872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15573803 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11589005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15563662 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59198822021239 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6745764 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58899933991936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180414464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59972697015808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6745767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56269427454976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58129440974704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1069467195001 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚1549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1072851044864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10662763 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11589005 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59972697015808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57373680610304 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15573803 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1104253155328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59972802076160zｓ0þ+ _wandbruntime‚596  _runtime‚596aÀã†É&ÿÃM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1073529161728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10669301 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60010741492736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58937300359168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3919247 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844975222422817e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27169886 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11596326 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11596328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3919501 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15573560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58937317398528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58937317398528 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27180043 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1073529161728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚1549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1105129845248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15583715 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15583715 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58937300359168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11596328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1070056997979 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56305122612224 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60010846560256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60010846560256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11596326 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1104957826560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27169886 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15573560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196922368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57410080438784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58166313042464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172018688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59236370040443 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59236473228381 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6750054 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1073441133568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1073441133568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58166330081824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚123 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184šÐg…”€ÇM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27180043 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1073529161728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚1549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1105129845248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15583715 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15583715 VCDeviceStatsMonitor.on_trai1f>Õ9n_batch_end/active_bytes.large_pool.freed‚58937300359168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11596328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1070056997979 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56305122612224 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60010846560256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60010846560256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11596326 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1104957826560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27169886 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15573560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196922368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57410080438784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58166313042464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172018688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59236370040443 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59236473228381 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6750054 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1073441133568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1073441133568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58166330081824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚123 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1073529161728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10669301 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60010741492736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58937300359168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3919247 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844975222422817e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27169886 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11596326 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11596328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3919501 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15573560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58937317398528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58937317398528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11955055 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59193614970368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213706752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 614.752579163 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59973061402064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6960264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188803072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11955055 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1106788786688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1139215946240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61874953119232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚126 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61875058179584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60768269392896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16070555 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16060414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0²Ã'‹½O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61875058179584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61874953119232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28025612 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11000574 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61076358072473 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60768252353536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59193401263616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6960267 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4040307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11000326 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1106700765696 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4040062 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58054185317376 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28015469 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11955055 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59193614970368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213706752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 614.752579163 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59973061402064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6960264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188803072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11955055 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1106788786688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1139215946240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61874953119232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚126 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61875058179584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60768269392896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16070555 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16060414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0~óàÅÌ0Œ‚(:¥ …‹õÒ‰ªù+ disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0! disk.dm-1.out‚139.27490234375$ memory_percent‚99.88119602203369' proc.memory.availableMB‚ 19.46484375 network.recv‚223630( disk./.usagePercent‚75.9525519592951' disk.nvme0c0n1.out‚2211.70361328125% disk.nvme0n1.out‚2211.70361328125 disk.dm-1.in‚0 proc.memory.rssMB‚2821 cpu‚26.041545146280445 network.sent‚ 1.840909e+06% disk./.usageGB‚1166.0615539550781 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚327.52783203125" disk.dm-0.out‚2350.09130859375' proc.memory.percent‚17.218017578125 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚327.52783203125 disk.md0.out‚ 23.6328125 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17Ϥô‹‚(:„ …‹õÒåÁª, gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚196.167' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.389 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûV‚*Ž+ _wandbruntime‚615  _runtimeS’†¥‚615Ãnƒ{É&ÃM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11962378 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61912997596160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4042501 _step‚127 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11962376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6964557 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60805618720768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11962376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28032688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4042760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28032688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60009933469824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1107378875392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61913102663680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58089905378304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚1599 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28042845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11007055 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60805635760128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11962378 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16080467 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28042845 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1107466903552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1107466903552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58089880474624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61113906091789 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 615.009760117 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11007317 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7844975410987024e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1103886473387 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1140100078592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61113802903851 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16080467omAÍ&ÇM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28032688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60009933469824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1107378875392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61913102663680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58089905378304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚1599 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28042845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11007055 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60805635760128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11962378 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16080467 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28042845 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1107466903552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1107466903552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58089880474624 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61113906091789 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 615.009760117 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11007317 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7844975410987024e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1103886473387 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1140100078592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61113802903851 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16080467 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11962378 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61912997596160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4042501 _step‚127 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11962376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6964557 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60805618720768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11962376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28032688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4042760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ð‰Ü0‘+ _wandbruntime‚615  _runtime‚615Í€ýû’ÿ /loss/FRt1/detection_loss‚0.5768010934326513 1loss/SRqqt1/detection_loss‚3.3520377923960334 % _timestamp‚1.7844975410996654e+09 _runtime‚ 615.010278248 _step‚128 /loss/FRt1/assignment_loss‚3.606785417506189 1loss/SRqqt2/detection_loss‚0.8952373418780483 /loss/FBt1/detection_loss‚3.4749719234598713 /loss/FBt2/detection_loss‚2.1132550560720516 epoch‚1 1loss/FBt1/assignment_loss‚0.33774291399411666 /loss/FRt2/detection_loss‚1.3671713221232709 &loss/total_loss‚20.642285643399482 2loss/SRqqt2/assignment_loss‚1.1594863554130175 trainer/global_step‚1599 /loss/FRt2/assignment_loss‚2.212455753193182 2loss/SRqqt1/assignment_loss‚1.3405761182618638 1loss/FBt2/assignment_loss‚0.20576455566918847µ¼žé “„ /loss/FBt1/detection_loss‚3.4749719234598713 /loss/FBt2/detection_loss‚2.1132550560720516 epoch‚1 1loss/FBt1/assignment_loss‚0.33774291399411666 /loss/FRt2/detection_loss‚1.3671713221232709 &loss/total_loss‚20.642285643399482 2loss/SRqqt2/assignment_loss‚1.1594863554130175 trainer/global_step‚1599 /loss/FRt2/assignment_loss‚2.212455753193182 2loss/SRqqt1/assignment_loss‚1.3405761182618638 1loss/FBt2/assignment_loss‚0.20576455566918847 /loss/FRt1/detection_loss‚0.5768010934326513 1loss/SRqqt1/detection_loss‚3.3520377923960334 % _timestamp‚1.7844975410996654e+09 _runtime‚ 615.010278248 _step‚128 /loss/FRt1/assignment_loss‚3.606785417506189 1loss/SRqqt2/detection_loss‚0.8952373418780483€³Š÷Ê.”‚(:£ ”‹õÒœ—ð+% disk./.usageGB‚1165.9505882263184 disk.md0.out‚ 23.88671875" disk.dm-0.out‚2350.56005859375 disk.rbd0.out‚0 disk.nvme0n1.in‚0' proc.memory.availableMB‚ 19.48046875 proc.cpu.threads‚17) disk./.usagePercent‚75.94532409876686 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 cpu‚26.110166734220225% disk.nvme1c1n1.out‚327.8935546875 disk.md0.in‚0! disk.dm-1.out‚139.32568359375$ memory_percent‚99.88110065460205 proc.memory.rssMB‚2821' proc.memory.percent‚17.218017578125 network.sent‚ 1.892384e+06# disk.nvme1n1.out‚327.8935546875 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚2212.3935546875$ disk.nvme0n1.out‚2212.3935546875 network.recv‚2277556î%pš•‚(: ”‹õÒµŠ, gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.67' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.223333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b?Î0–+ _wandbruntime‚631  _runtime‚631C¼Ê7ç&—è{ =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 )jet/accuracy_1_of_2‚0.668180525302887 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5631888508796692 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49774178862571716 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13666851818561554 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4835379123687744 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07423349469900131 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.32338762283325195 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24387668073177338 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9600468873977661 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013896243646740913 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0723297968506813 _runtime‚ 631.058879352 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2660260498523712 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30951863527297974 'particle/f_score‚0.5672760605812073 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.762499988079071 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6964285969734192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9664613008499146 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09810062497854233 *jet/accuracy_2_of_4‚0.2102300524711609 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.596855878829956 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16125676035881042 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5348155498504639 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4162955582141876 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45459094643592834 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10758568346500397 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1403825730085373 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3499999940395355 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29627755284309387 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9009239673614502 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9689818620681763 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12159112840890884 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7099999785423279 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08107596635818481 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.590413510799408 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 +jet/accuracy_2_of_3‚0.24840576946735382 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9616213440895081 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10533930361270905 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.5504786968231201 7validation_average_jet_accuracy‚0.26250213384628296 % _timestamp‚1.7844975571475086e+09 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03655879572033882 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4127650856971741 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.21171875298023224 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09675557166337967 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19349431991577148 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6323904395103455 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1269117146730423 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8245156407356262 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07778487354516983 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5556157231330872 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4285714328289032 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9896187782287598 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39465558528900146 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 _step‚129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2765151262283325 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9183760285377502 trainer/global_step‚1601 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4081196188926697 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2720169425010681 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1061118021607399 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18833692371845245 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.30654722452163696 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49380868673324585 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.997619092464447 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2720169425010681 particle/accuracy_2_of_2‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 -jet/accuracy_4_of_4‚0.0059523810632526875 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2847222089767456 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9843387603759766 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09416328370571136 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6307017207145691 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13483141362667084 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6922966241836548 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.22233302891254425 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34240031242370605 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.29882943630218506 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019757255911827087 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.15261971950531006 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1403825730085373 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4548065960407257 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.8964347839355469 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09810062497854233 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.32334157824516296 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6481263041496277 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42620664834976196 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4731156826019287 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45459094643592834 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9777072668075562 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9727599024772644 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5543616414070129 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005897243041545153 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1167609691619873 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0891302302479744 ,jet/accuracy_3_of_4‚0.029232805594801903 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3747236430644989 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7099999785423279 +particle/specificity‚0.8932814002037048 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0833333358168602 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_puz÷®rity‚0.5222039818763733 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2808642089366913 *jet/accuracy_1_of_4‚0.6854031085968018 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4564839005470276 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12834560871124268 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011363514699041843 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9896187782287598 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3360886871814728 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900132417678833 /particle/accuracy_3_of_3‚0.9994708299636841 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.8973461389541626 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26358938217163086 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.16587145626544952 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9088015556335449 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5003265142440796 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02708333358168602 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.1599999964237213 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2619265019893646 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03578563779592514 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.555924117565155 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007465593051165342 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5469840168952942 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.1428571492433548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7450905442237854 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.45644205808639526 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.42818769812583923 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9943783283233643 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.500687837600708 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3005914092063904 *jet/accuracy_1_of_3‚0.7653613090515137 jet/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9847977161407471 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931069612503052 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9912216663360596 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1937102973461151 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.159539595246315 ,jet/accuracy_3_of_3‚0.036113984882831573 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8825463056564331 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08066896349191666 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6371793746948242 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.403920978307724 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24098850786685944 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2966376841068268 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 epoch‚1 *loss/val_total_loss‚19.644419053757595 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9926981925964355 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19949567317962646 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15938889980316162 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49868693947792053 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3058081567287445 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08025120198726654 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2954545319080353 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/accuracy‚0.819669246673584 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4456981420516968 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.30917230248451233 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19748201966285706 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3638209104537964 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29249587655067444 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4394736588001251 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2428896129131317 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.7928694486618042 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003872779430821538 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9930699467658997 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9518016576766968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9315789937973022 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7899127006530762 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9902019500732422 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5013734102249146Ø„“Hó=˜í{ @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13483141362667084 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6922966241836548 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.22233302891254425 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34240031242370605 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.29882943630218506 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019757255911827087 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.15261971950531006 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1403825730085373 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4548065960407257 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.8964347839355469 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09810062497854233 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.32334157824516296 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6481263041496277 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42620664834976196 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4731156826019287 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45459094643592834 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9777072668075562 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9727599024772644 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5543616414070129 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005897243041545153 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1167609691619873 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0891302302479744 ,jet/accuracy_3_of_4‚0.029232805594801903 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3747236430644989 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7099999785423279 +particle/specificity‚0.8932814002037048 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0833333358168602 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5222039818763733 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2808642089366913 *jet/accuracy_1_of_4‚0.6854031085968018 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4564839005470276 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12834560871124268 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011363514699041843 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9896187782287598 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3360886871814728 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900132417678833 /particle/accuracy_3_of_3‚0.9994708299636841 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.8973461389541626 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26358938217163086 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.16587145626544952 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9088015556335449 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5003265142440796 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02708333358168602 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.1599999964237213 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2619265019893646 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03578563779592514 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.555924117565155 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007465593051165342 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5469840168952942 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.1428571492433548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7450905442237854 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.45644205808639526 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.42818769812583923 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9943783283233643 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.500687837600708 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3005914092063904 *jet/accuracy_1_of_3‚0.7653613090515137 jet/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9847977161407471 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931069612503052 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9912216663360596 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1937102973461151 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.159539595246315 ,jet/accuracy_3_of_3‚0.036113984882831573 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8825463056564331 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08066896349191666 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6371793746948242 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.403920978307724 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24098850786685944 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2966376841068268 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 epoch‚1 *loss/val_total_loss‚19.644419053757595 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9926981925964355 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19949567317962646 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15938889980316162 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49868693947792053 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3058081567287445 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08025120198726654 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2954545319080353 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/accuracy‚0.819669246673584 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4456981420516968 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.30917230248451233 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19748201966285706 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3638209104537964 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29249587655067444 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4394736588001251 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2428896129131317 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.7928694486618042 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003872779430821538 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9930699467658997 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9518016576766968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9315789937973022 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7899127006530762 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9902019500732422 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5013734102249146 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 )jet/accuracy_1_of_2‚0.668180525302887 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5631888508796692 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49774178862571716 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13666851818561554 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4835379123687744 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07423349469900131 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.32338762283325195 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24387668073177338 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9600468873977661 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013896243646740913 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0723297968506813 _runtime‚ 631.058879352 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2660260498523712 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30951863527297974 'particle/f_score‚0.5672760605812073 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.762499988079071 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6964285969734192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9664613008499146 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09810062497854233 *jet/accuracy_2_of_4‚0.2102300524711609 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.596855878829956 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16125676035881042 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5348155498504639 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4162955582141876 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45459094643592834 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10758568346500397 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1403825730085373 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3499999940395355 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29627755284309387 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9009239673614502 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9689818620681763 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12159112840890884 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7099999785423279 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08107596635818481 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.590413510799408 particle/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 +jet/accuracy_2_of_3‚0.24840576946735382 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9616213440895081 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10533930361270905 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.5504786968231201 7validation_average_jet_accuracy‚0.26250213384628296 % _timestamp‚1.7844975571475086e+09 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03655879572033882 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4127650856971741 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.21171875298023224 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09675557166337967 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19349431991577148 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6323904395103455 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1269117146730423 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8245156407356262 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07778487354516983 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5556157231330872 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4285714328289032 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9896187782287598 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39465558528900146 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 _step‚129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2765151262283325 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9183760285377502 trainer/global_step‚1601 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4081196188926697 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2720169425010681 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1061118021607399 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18833692371845245 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.30654722452163696 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49380868673324585 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.997619092464447 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2720169425010681 particle/accuracy_2_of_2‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 -jet/accuracy_4_of_4‚0.0059523810632526875 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2847222089767456 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9843387603759766 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09416328370571136 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6307017207145691xüLS-™‚(:¢ £‹õÒíߤ+! disk.nvme1c1n1.out‚ 392.421875) disk./.usagePercent‚75.95347205856068 disk.md0.out‚ 24.1953125 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚2216.8779296875 disk.nvme1c1n1.in‚0$ memory_percent‚99.20389652252197* proc.memory.percent‚17.218542098999023& disk.nvme0c0n1.out‚2216.8779296875( proc.memory.availableMB‚ 130.43359375" proc.memory.rssMB‚ 2821.0859375 proc.cpu.threads‚17 network.sent‚ 1.931455e+06 network.recv‚232592 disk.nvme1n1.in‚0 disk.md0.in‚0! disk.dm-0.out‚2354.9755859375 cpu‚26.109604112424996% disk./.usageGB‚1166.0756797790527 disk.nvme1n1.out‚ 392.421875 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 203.25390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0Z¼Ïz›š‚(: £‹õÒµú+ gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.611' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.20366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†°†0›+ _wandbruntime‚649  _runtime‚649ö2©ÆœÀ _runtime‚ 649.459381138 _step‚130 & lr-AdamW/pg1‚0.0001016402695754414 & lr-AdamW/pg2‚0.0001016402695754414 trainer/global_step‚1649 % _timestamp‚1.7844975755474494e+09£‹oúËÅ _step‚130 & lr-AdamW/pg1‚0.0001016402695754414 & lr-AdamW/pg2‚0.0001016402695754414 trainer/global_step‚1649 % _timestamp‚1.7844975755474494e+09 _runtime‚ 649.459381138‚6jš0ž+ _wandbruntime‚649  _runtime‚649µó\$Ÿ¹O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29283000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11514906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62785154258416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172025344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1157434004909 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196929024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16724427 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1161146843648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29293144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1157520153379 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64770293132288 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12558575 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7268847 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64770398193152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63609251349504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12558575 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16724427 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12558573 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7844975755480225e+09 _step‚131 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63609234310144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1161058822144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4246062 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63609251349504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1194723062272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12558573 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61959089505792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60764366443520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61959286434816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16734569 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1161146843648 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1161058822144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1194895087616 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63942674411795 _runtime‚ 649.459446218 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4246287 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142Ièî-Ä' ¾O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚228 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7268844 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64770398193152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1161058822144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1194895087616 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63942674411795 _runtime‚ 649.459446218 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4246287 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29283000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11514906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62785154258416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172025344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1157434004909 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196929024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16724427 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1161146843648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29293144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1157520153379 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64770293132288 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12558575 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7268847 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64770398193152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63609251349504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12558575 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16724427 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12558573 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7844975755480225e+09 _step‚131 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63609234310144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1161058822144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4246062 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63609251349504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1194723062272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12558573 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61959089505792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60764366443520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61959286434816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16734569 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1161146843648 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1158109956353 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1161824960512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29300219 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7273137 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4248725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4248484 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1195600792064 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29310377 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29300219 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11521618 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62822009286816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62822026326176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63646600677376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1161824960512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29310377 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12565894 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16734325 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63980222431111 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63646617716736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61995687296512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1158196104935 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1161736931840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16744481 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚132 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60800086504448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63980119243169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64808337609216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1649 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64808442677248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12565896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16734325Þá Í&£ÇM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1158109956353 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1161824960512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29300219 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7273137 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4248725 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4248484 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1195600792064 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29310377 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29300219 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11521618 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62822009286816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62822026326176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63646600677376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1161824960512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29310377 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12565894 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16734325 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63980222431111 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63646617716736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61995687296512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1158196104935 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1161736931840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16744481 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚132 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60800086504448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63980119243169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64808337609216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1649 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64808442677248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12565896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16734325 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65477569711104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30155946 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17221179 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1195084585472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11852344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1699 % _timestamp‚1.7844975944941812e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65477552671744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65477569711104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17231321 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66672549235712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7483344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63778837479424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12924625 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11852096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17221179 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1191349628787 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1194996563968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65477552671744 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7483347 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1229897781760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64628757646416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30145802 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12924623 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17231321 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65820107275203 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66672549235712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373DѵÄ'®¾O N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17221179 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1191349628787 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1194996563968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65477552671744 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7483347 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1229897781760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64628757646416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30145802 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12924623 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17231321 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65820107275203 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66672549235712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62549149209600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1191263480317 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12924623 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65477569711104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30155946 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17221179 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1195084585472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11852344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1699 % _timestamp‚1.7844975944941812e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65477552671744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65477569711104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17231321 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66672549235712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7483344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63778837479424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12924625 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11852096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30173179 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65514919038976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65857655294519 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1195674673664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62584819463168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30173179 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63815229455872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11858788 % _timestamp‚1.7844975947531776e+09 _step‚136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65514936078336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12931944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7487637 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4371154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12931946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17241233 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12931946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11859035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 trainer/global_step‚1699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1195762702336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230598788096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230409992704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64665612674816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1191939431761 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66710698780672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66710593712640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64665629714176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4371398‘|ıÈM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30163021 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30163021 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30173179 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65514919038976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65857655294519 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1195674673664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62584819463168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30173179 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63815229455872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11858788 % _timestamp‚1.7844975947531776e+09 _step‚136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65514936078336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12931944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7487637 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4371154 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12931946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17241233 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12931946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11859035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 trainer/global_step‚1699 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1195762702336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230598788096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1230409992704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64665612674816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1191939431761 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66710698780672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66710593712640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64665629714176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4371398 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17231077 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65857552106577 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747ˆ£ƒ0²+ _wandbruntime‚668  _runtime‚668%š²³ü &loss/total_loss‚20.074548152857837 epoch‚2 % _timestamp‚1.7844975947541115e+09 /loss/FRt1/assignment_loss‚3.493502846344765 0loss/FBt1/assignment_loss‚0.3196080301559841 1loss/FBt2/assignment_loss‚0.19794015656719108 /loss/FRt2/detection_loss‚1.3156337270088856 _runtime‚ 668.665140187 /loss/FRt2/assignment_loss‚2.203595946796916 1loss/SRqqt1/assignment_loss‚1.323160222104072 2loss/SRqqt2/assignment_loss‚1.1518088528173394 1loss/SRqqt2/detection_loss‚0.8725767931933888 trainer/global_step‚1699 _step‚137 /loss/FRt1/detection_loss‚0.5797608854625208 1loss/SRqqt1/detection_loss‚2.2527630010130673 /loss/FBt1/detection_loss‚3.8209381669694107 .loss/FBt2/detection_loss‚2.543259524424294‘dó´ &loss/total_loss‚20.074548152857837 epoch‚2 % _timestamp‚1.7844975947541115e+09 /loss/FRt1/assignment_loss‚3.493502846344765 0loss/FBt1/assignment_loss‚0.3196080301559841 1loss/FBt2/assignment_loss‚0.19794015656719108 /loss/FRt2/detection_loss‚1.3156337270088856 _runtime‚ 668.665140187 /loss/FRt2/assignment_loss‚2.203595946796916 1loss/SRqqt1/assignment_loss‚1.323160222104072 2loss/SRqqt2/assignment_loss‚1.1518088528173394 1loss/SRqqt2/detection_loss‚0.8725767931933888 trainer/global_step‚1699 _step‚137 /loss/FRt1/detection_loss‚0.5797608854625208 1loss/SRqqt1/detection_loss‚2.2527630010130673 /loss/FBt1/detection_loss‚3.8209381669694107 .loss/FBt2/detection_loss‚2.543259524424294‰ñbI6µ‚(:« Á‹õÒðüÒ+ network.recv‚241755) disk./.usagePercent‚75.95347230703469# disk./.usageGB‚1166.07568359375 disk.dm-1.in‚0 disk.dm-1.out‚267.4619140625 disk.md0.out‚ 24.8046875 proc.cpu.threads‚17" disk.dm-0.out‚2357.03466796875 disk.nvme0n1.in‚0% disk.nvme1c1n1.out‚457.5224609375 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚2219.37646484375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚457.5224609375 disk.md0.in‚0& proc.memory.availableMB‚ 130.296875" proc.memory.rssMB‚ 2821.0859375 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2219.37646484375$ memory_percent‚99.20473098754883* proc.memory.percent‚17.218542098999023 cpu‚26.102253668522447 network.sent‚ 2.050753e+068y[Λ¶‚(: Á‹õÒ┩, gpu.0.gpu‚79.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚178.658' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚59.55266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥×4@0·+ _wandbruntime‚687  _runtime‚687œó)¸Ã¸½ ! _timestamp‚1.784497613389e+09 _runtime‚ 687.29992688 _step‚138 ' lr-AdamW/pg1‚0.00010161102267618607 ' lr-AdamW/pg2‚0.00010161102267618607 trainer/global_step‚1749úO$ªÈ¹Â trainer/global_step‚1749 ! _timestamp‚1.784497613389e+09 _runtime‚ 687.29992688 _step‚138 ' lr-AdamW/pg1‚0.00010161102267618607 ' lr-AdamW/pg2‚0.00010161102267618607Š¿ òª0º+ _wandbruntime‚687  _runtime‚687FJ[V¿'»¹O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17728073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17717931 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65598531922432 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67345871033344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66472343995056 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65598737240064 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264649754112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 687.300749952 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31008604 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4492612 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67697540138611 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67345888072704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17717931 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64333882168320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7697844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66472361034416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12190707 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68574910400000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1229022327296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12190456 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1225179104195 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1229022327296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7844976133899167e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68574805339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4492860 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1228934305792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290675 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31018748 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290673 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7697847 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413952 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64333907072000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67697436950781 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68574910400000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1228934305792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68574910400000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1228934305792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264649754112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 687.300749952 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31008604 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4492612 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67697540138611 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67345888072704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17717931 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64333882168320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7697844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66472361034416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12190707 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68574910400000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1229022327296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12190456 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1225179104195 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1229022327296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7844976133899167e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68574805339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4492860 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1228934305792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290675 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31018748 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290673 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7697847 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413952 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64333907072000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67697436950781 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31035981 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚140 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68612954884096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12197232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67735088157927 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13297994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1225855055751 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1229612415488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64369602229248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1229612415488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297996 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17737985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31025823 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67383254439936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1229700444160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1229700444160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17737985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64369577325568 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297996 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1225768907169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297994 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17727829 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31035981 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0½ ¿ÆÍ&¿ÇM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1225855055751 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1229612415488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64369602229248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1229612415488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297996 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17737985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31025823 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67383254439936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1229700444160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1229700444160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17737985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64369577325568 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297996 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1225768907169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297994 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17727829 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31035981 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31035981 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚140 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68612954884096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12197232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67735088157927 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13297994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Œ´¾Òb0À+ _wandbruntime‚687  _runtime‚687Ÿ5:~Áþ 0loss/FRt1/assignment_loss‚3.5296896904858235 1loss/FBt1/assignment_loss‚0.22838177460623746 1loss/FBt2/assignment_loss‚0.11579176774331228 /loss/FBt1/detection_loss‚3.9801669757118003 &loss/total_loss‚19.665679391757095 2loss/SRqqt1/assignment_loss‚1.6637820782622805 /loss/FRt1/detection_loss‚0.6061801319232158 /loss/FRt2/detection_loss‚1.4614597181658306 epoch‚2 _runtime‚ 687.558357251 _step‚141 0loss/SRqqt1/detection_loss‚2.228936981276126 /loss/FRt2/assignment_loss‚2.223567355811065 1loss/SRqqt2/assignment_loss‚1.117768045194447 1loss/SRqqt2/detection_loss‚0.7586902197822797 /loss/FBt2/detection_loss‚1.7512646527946785 trainer/global_step‚1749 % _timestamp‚1.7844976136477365e+09Jz) ƒ 1loss/FBt2/assignment_loss‚0.11579176774331228 /loss/FBt1/detection_loss‚3.9801669757118003 &loss/total_loss‚19.665679391757095 2loss/SRqqt1/assignment_loss‚1.6637820782622805 /loss/FRt1/detection_loss‚0.6061801319232158 /loss/FRt2/detection_loss‚1.4614597181658306 epoch‚2 _runtime‚ 687.558357251 _step‚141 0loss/SRqqt1/detection_loss‚2.228936981276126 /loss/FRt2/assignment_loss‚2.223567355811065 1loss/SRqqt2/assignment_loss‚1.117768045194447 1loss/SRqqt2/detection_loss‚0.7586902197822797 /loss/FBt2/detection_loss‚1.7512646527946785 trainer/global_step‚1749 % _timestamp‚1.7844976136477365e+09 0loss/FRt1/assignment_loss‚3.5296896904858235 1loss/FBt1/assignment_loss‚0.22838177460623746¬]Äû3Â(:¨ ЋõÒãðÉ+ disk.rbd0.out‚0& disk.nvme1c1n1.out‚458.43310546875$ disk.nvme1n1.out‚458.43310546875 disk.dm-0.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.md0.out‚25.28125( proc.memory.availableMB‚ 130.25390625 cpu‚26.09794419812111 network.sent‚ 2.12685e+06) disk./.usagePercent‚75.77646712374754% disk.nvme0c0n1.out‚2220.392578125# disk.nvme0n1.out‚2220.392578125 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-0.out‚2357.89794921875 disk.dm-1.in‚0$ disk./.usageGB‚1163.358211517334 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.dm-1.out‚267.5009765625$ memory_percent‚99.20499324798584" proc.memory.rssMB‚ 2821.0859375* proc.memory.percent‚17.218542098999023 network.recv‚246180 disk.rbd0.in‚0Ϊºž›Ä‚(: ЋõÒô±ó+ gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.19566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h¨*:Å‚(:¯ ß‹õÒãÑ»+$ memory_percent‚99.20506477355957 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚2224.01611328125& disk.nvme1c1n1.out‚459.21630859375! disk.dm-0.out‚2361.2080078125 disk.rbd0.out‚0* proc.memory.percent‚17.218542098999023 cpu‚26.095210529919456 network.sent‚ 2.177089e+06 network.recv‚250573' disk.nvme0c0n1.out‚2224.01611328125 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 267.689453125" proc.memory.rssMB‚ 2821.0859375% disk./.usageGB‚1163.3583526611328 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚459.21630859375 disk.md0.out‚ 25.69921875 disk.dm-0.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 130.2421875) disk./.usagePercent‚75.776476317285978ú‡šÆ‚(: ß‹õÒ´Éæ+ gpu.0.gpu‚32.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚143.096' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚47.69866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Ì0Ç+ _wandbruntime‚706  _runtime‚706˜ð’²ÈÈ ' lr-AdamW/pg1‚0.00010159518023484611 ' lr-AdamW/pg2‚0.00010159518023484611 trainer/global_step‚1799 % _timestamp‚1.7844976322264469e+09 _runtime‚ 706.137345292 _step‚1425ccÍÉÇ ' lr-AdamW/pg1‚0.00010159518023484611 ' lr-AdamW/pg2‚0.00010159518023484611 trainer/global_step‚1799 % _timestamp‚1.7844976322264469e+09 _runtime‚ 706.137345292 _step‚142Ž úõ&0Ê+ _wandbruntime‚706  _runtime‚706yÙÃ5!˹O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69214189394944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18224825 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69574869814189 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1262960069120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13656725 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 % _timestamp‚1.7844976322270832e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31881550 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4615507 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4615268 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1262960069120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1262872047616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1262872047616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67418419730944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7912347 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13656723 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18214683 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67418216510464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 706.139321555 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70477061442560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1799 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18214683 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13656725 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68315964422416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚143 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70477166503424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70477166503424 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70477061442560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18224825 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12527854 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12527612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203220480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69214206434304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31871406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299576479744 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66118640030720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69214206434304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69574973002019 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1258922431133 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1259008579603 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712dÙܸÄ'̾O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203220480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69214206434304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31871406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299576479744 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66118640030720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69214206434304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69574973002019 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1258922431133 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1259008579603 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69214189394944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18224825 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69574869814189 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1262960069120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13656725 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 % _timestamp‚1.7844976322270832e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31881550 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4615507 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4615268 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1262960069120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1262872047616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1262872047616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67418419730944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7912347 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13656723 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18214683 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67418216510464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 706.139321555 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70477061442560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1799 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18214683 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13656725 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68315964422416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚143 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70477166503424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70477166503424 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70477061442560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18224825 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12527854 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12527612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70515105919488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69251555762176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68352836490176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚1799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1300461839360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67454604037632 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13664044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69251572801536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12534631 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69251572801536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70515210987520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12534375 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 % _timestamp‚1.7844976324846494e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69251555762176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4617741 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1263638185984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1263638185984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66154335187968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31898783 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31888625 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18234737 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69612521021335 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1263550157312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4617994 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1263550157312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217893376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70515210987520 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31898783 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31888625 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7916637 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66154360091648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18224581 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1259684531159 _runtime‚ 706.395762884 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1300268849664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1259598382577 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18234737 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0úö Î&ÏÈM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13664044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69251572801536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12534631 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69251572801536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70515210987520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12534375 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 % _timestamp‚1.7844976324846494e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69251555762176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4617741 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1263638185984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1263638185984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66154335187968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31898783 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31888625 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18234737 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69612521021335 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1263550157312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4617994 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1263550157312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217893376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70515210987520 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31898783 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31888625 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7916637 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66154360091648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18224581 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1259684531159 _runtime‚ 706.395762884 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1300268849664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1259598382577 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18234737 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69612417833393 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70515105919488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69251555762176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68352836490176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚1799 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1300461839360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67454604037632|(0Ð+ _wandbruntime‚706  _runtime‚706nâÿÚvÑÿ 0loss/FRt2/assignment_loss‚1.9872205240493317 2loss/SRqqt1/assignment_loss‚1.4964712726017624 .loss/FRt1/detection_loss‚0.509809105275774 _runtime‚ 706.396190834 2loss/SRqqt2/assignment_loss‚1.0353080849796428 1loss/FBt1/assignment_loss‚0.31173408437364725 1loss/FBt2/assignment_loss‚0.17426420632737064 1loss/SRqqt1/detection_loss‚2.0Ž\ëÔ670450547260337 /loss/FBt1/detection_loss‚2.5973203297524043 .loss/FBt2/detection_loss‚1.880367133372448 epoch‚2 trainer/global_step‚1799 /loss/FRt2/detection_loss‚1.4083483006202173 1loss/SRqqt2/detection_loss‚0.8518248376008275 &loss/total_loss‚17.748024247878817 0loss/FRt1/assignment_loss‚3.4283113141993544 % _timestamp‚1.7844976324855728e+09 _step‚145Â,c Ò„ .loss/FRt1/detection_loss‚0.509809105275774 _runtime‚ 706.396190834 2loss/SRqqt2/assignment_loss‚1.0353080849796428 1loss/FBt1/assignment_loss‚0.31173408437364725 1loss/FBt2/assignment_loss‚0.17426420632737064 1loss/SRqqt1/detection_loss‚2.0670450547260337 /loss/FBt1/detection_loss‚2.5973203297524043 .loss/FBt2/detection_loss‚1.880367133372448 epoch‚2 trainer/global_step‚1799 /loss/FRt2/detection_loss‚1.4083483006202173 1loss/SRqqt2/detection_loss‚0.8518248376008275 &loss/total_loss‚17.748024247878817 0loss/FRt1/assignment_loss‚3.4283113141993544 % _timestamp‚1.7844976324855728e+09 _step‚145 0loss/FRt2/assignment_loss‚1.9872205240493317 2loss/SRqqt1/assignment_loss‚1.4964712726017624‘RE°,,Ó‚(:¡ î‹õÒ¦ãÚ+ network.recv‚253987! disk.dm-0.out‚2362.1337890625 disk.dm-1.in‚0% disk./.usageGB‚1163.1216049194336 disk.dm-0.in‚0 disk.dm-1.out‚ 267.814453125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 459.9296875 disk.md0.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚2224.8896484375 disk.md0.out‚ 25.9453125# memory_percent‚99.2070198059082& proc.memory.availableMB‚ 129.921875" proc.memory.rssMB‚ 2821.0859375 cpu‚26.091880192830548 proc.cpu.threads‚17 network.sent‚ 2.192997e+06( disk./.usagePercent‚75.7610555231678 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚2224.8896484375" disk.nvme1c1n1.out‚ 459.9296875* proc.memory.percent‚17.218542098999023GΗ¢šÔ‚(: î‹õÒ§šƒ, gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚145.243' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚48.41433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A¦]0Õ+ _wandbruntime‚724  _runtime‚724CöqÂÇÖÁ ' lr-AdamW/pg1‚0.00010157852546635806 ' lr-AdamW/pg2‚0.00010157852546635806 trainer/global_step‚1849 $ _timestamp‚1.784497650993675e+09 _runtime‚ 724.904631685 _step‚14691ß›Ì×Æ ' lr-AdamW/pg2‚0.00010157852546635806 trainer/global_step‚1849 $ _timestamp‚1.784497650993675e+09 _runtime‚ 724.904631685 _step‚146 ' lr-AdamW/pg1‚0.00010157852546635806’6`·0Ø+ _wandbruntime‚724  _runtime‚724c)ú¿'Ù¹O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4738301 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚147 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚1849 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1296809789440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32744352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72379317545984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32734208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1296897810944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71452405865427 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70159567810416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14022775 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72379317545984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67903422796800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71082524795904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4738049 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1334707406336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70159550771056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1296809789440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1292838055011 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18721577 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12865148 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67903397893120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14022773 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8126847 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1292751906541 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18711435 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _runtime‚ 724.909405683 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71452302677597 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18721577 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 % _timestamp‚1.7844976509946265e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1296897810944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69237920691200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72379422606848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72379317545984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32734208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1296897810944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71452405865427 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70159567810416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14022775 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72379317545984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67903422796800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71082524795904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4738049 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1334707406336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70159550771056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1296809789440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1292838055011 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18721577 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12865148 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67903397893120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14022773 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8126847 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1292751906541 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18711435 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _runtime‚ 724.909405683 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71452302677597 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18721577 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 % _timestamp‚1.7844976509946265e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1296897810944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69237920691200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72379422606848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72417467090944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32761585 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14030094 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70196422838816 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71489953884743 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18721333 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1293427857985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1297487899136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69274523208704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 % _timestamp‚1.7844976512520626e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71119891163136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32751427 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1293514006567 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71119891163136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67939093050368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12871687 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1297575927808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1297487899136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14030096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71119874123776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1335224847872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚148 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72417362022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70196439878176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8131134 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32751427 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310464 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1335224847872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚148 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72417362022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70196439878176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8131134 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32751427 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310464 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72417467090944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32761585 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14030094 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70196422838816 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71489953884743 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18721333 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1293427857985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1297487899136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69274523208704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 % _timestamp‚1.7844976512520626e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71119891163136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32751427 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1293514006567 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71119891163136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67939093050368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12871687 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1297575927808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1297487899136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14030096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0”Ô»â»0Þ+ _wandbruntime‚725  _runtime‚725éD ÿßù &loss/total_loss‚17.545029021680627 0loss/FRt1/assignment_loss‚3.4276246809731803 2loss/SRqqt1/assignment_loss‚1.3468539595327995 /loss/SRqqt1/detection_loss‚2.14695930788183 trainer/global_step‚1849 0loss/FRt2/assignment_loss‚2.0432966533322006 _step‚149 epoch‚2 1loss/SRqqt2/assignment_loss‚1.141737259966858 0loss/FBt1/assignment_loss‚0.2842413267093804 .loss/FRt2/detection_loss‚1.367669745536562 0loss/SRqqt2/detection_loss‚0.855120375036567 /loss/FBt1/detection_loss‚1.3049463092285383 % _timestamp‚1.7844976512529998e+09 1loss/FBt2/assignment_loss‚0.04330871026122036 .loss/FRt1/detection_loss‚0.575725500705832 .loss/FBt2/detection_loss‚3.007545192515658 _runtime‚ 725.163659331chEàþ &loss/total_loss‚17.545029021680627 0loss/FRt1/assignment_loss‚3.4276246809731803 2loss/SRqqt1/assignment_loss‚1.3468539595327995 /loss/SRqqt1/detection_loss‚2.14695930788183 trainer/global_step‚1849 0loss/FRt2/assignment_loss‚2.0432966533322006 _step‚149 epoch‚2 1loss/SRqqt2/assignment_loss‚1.141737259966858 0loss/FBt1/assignment_loss‚0.2842413267093804 .loss/FRt2/detection_loss‚1.367669745536562 0loss/SRqqt2/detection_loss‚0.855120375036567 /loss/FBt1/detection_loss‚1.3049463092285383 % _timestamp‚1.7844976512529998e+09 1loss/FBt2/assignment_loss‚0.04330871026122036 .loss/FRt1/detection_loss‚0.575725500705832 .loss/FBt2/detection_loss‚3.007545192515658 _runtime‚ 725.163659331•žˆ0á‚(:¥ ý‹õÒÎì+ disk.dm-0.in‚0 disk.dm-1.in‚0 cpu‚26.089025185283386 network.recv‚258156 disk.nvme0n1.in‚0# disk.nvme0n1.out‚2233.482421875 disk.nvme1n1.in‚0# disk.nvme1n1.out‚461.9052734375 disk.md0.out‚ 26.390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk./.usageGB‚1163.1220092773438 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚461.9052734375 disk.md0.in‚0! disk.dm-0.out‚2371.6162109375 disk.dm-1.out‚ 267.9609375" proc.memory.rssMB‚ 2821.0859375* proc.memory.percent‚17.218542098999023 network.sent‚ 2.255123e+06 disk.rbd0.in‚0% disk.nvme0c0n1.out‚2233.482421875$ memory_percent‚99.20573234558105' proc.memory.availableMB‚ 130.1328125 proc.cpu.threads‚17) disk./.usagePercent‚75.76108186141305pƉ|›â‚(: ý‹õÒ…‡”, gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚122.382' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.794000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„£Œ”0ã+ _wandbruntime‚743  _runtime‚743ºWgËÇäÁ ' lr-AdamW/pg2‚0.00010156105863748183 trainer/global_step‚1899 $ _timestamp‚1.784497669396972e+09 _runtime‚ 743.308487873 _step‚150 ' lr-AdamW/pg1‚0.00010156105863748183‘—k|ÌåÆ ' lr-AdamW/pg1‚0.00010156105863748183 ' lr-AdamW/pg2‚0.00010156105863748183 trainer/global_step‚1899 $ _timestamp‚1.784497669396972e+09 _runtime‚ 743.308487873 _step‚150–4ƒ`ƒ0æ+ _wandbruntime‚743  _runtime‚7439žŠ­¿'ç¹O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1326581381949 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19218329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72950826118144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71057902639616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74281573649408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14388823 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74281678710272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72950843157504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13203356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33597010 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1330747531264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19218329 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚1899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19208187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19208187 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4861771 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14388825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72950826118144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69688155755520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73329838728835 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1330835552768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4862009 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 743.308549143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚151 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14388823 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69688180659200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72003154159056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72003171198416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8341347 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8341344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71057695224832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74281678710272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74281573649408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1369721980416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1326667530419 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1369539469312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1330835552768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Ñy² "è¾O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8341347 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8341344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71057695224832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74281678710272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74281573649408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1369721980416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1326667530419 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1369539469312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1330835552768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1326581381949 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19218329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72950826118144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71057902639616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74281573649408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14388823 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74281678710272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72950843157504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13203356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33597010 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1330747531264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19218329 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚1899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19208187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19208187 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4861771 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14388825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72950826118144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69688155755520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73329838728835 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 GADeviceStatsMonitor.on_traT^Ù»in_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1330835552768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4862009 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 743.308549143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚151 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14388823 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69688180659200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72003154159056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72003171198416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3—äâT»0é+ _wandbruntime‚743  _runtime‚743EòãÉ&êÃM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33614229 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73367386748151 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1327343481975 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33624387 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33624387 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72040043266176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72988209524736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74319723194368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72040026226816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1331425640960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13209835 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7844976696522574e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14396144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74319723194368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1370420033536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14396146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1331513669632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14396144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19228241 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4864439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74319618126336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19218085 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4864201 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19218085 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72988192485376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71094295849984 _step‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19228241 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1331513669632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72988209524736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1331513669632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72988209524736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73367386748151 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1327343481975 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33624387 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33624387 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72040043266176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72988209524736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74319723194368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72040026226816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1331425640960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13209835 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7844976696522574e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14396144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74319723194368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1370420033536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14396146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1331513669632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14396144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19228241 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4864439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74319618126336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19218085 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4864201 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19218085 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72988192485376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71094295849984 _step‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19228241 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75207271592243 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74819161519104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14754873 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14754873 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1404643495424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73846774586416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1364773294592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72877370407936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8555844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73846757547056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1364685273088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1364773294592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13542006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71472913617920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784497688041426e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75207168404413 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74819144479744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 _step‚155 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76183829752832 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72877582017024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1404456790016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74819144479744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19704939 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1364685273088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76183829752832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14754875 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76183934813696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19715081 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4985910 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34469956 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1360497005827 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74819161519104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71472938521600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75207271592243 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74819161519104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14754873 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14754873 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1404643495424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73846774586416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1364773294592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72877370407936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8555844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73846757547056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1364685273088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1364773294592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13542006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71472913617920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784497688041426e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75207168404413 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74819144479744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 _step‚155 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76183829752832 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72877582017024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1404456790016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74819144479744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19704939 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1364685273088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76183829752832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14754875 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76183934813696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19715081 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4985910 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34469956›„…!“0÷+ _wandbruntime‚762  _runtime‚762}ý#hÉ&øÃM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1949 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74856510846976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19724993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4988582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74856510846976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19714837 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74856527886336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1365451411456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1365363382784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34487189 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34477031 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34487189 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13548719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13548471 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1405340192768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7844976882964895e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34477031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73883629614816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1361086808801 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72913973871616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71508633678848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1405153494528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14762194 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14762196 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75244716423617 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74856527886336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361172957383 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19724993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19714837 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1365451411456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1365363382784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 _runtime‚ 762.207519935 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71508608775168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14762196 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76221979297792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0À=ïPÎ&ùÈM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73883629614816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1361086808801 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72913973871616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71508633678848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1405153494528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14762194 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14762196 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75244716423617 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74856527886336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361172957383 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19724993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19714837 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1365451411456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1365363382784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 _runtime‚ 762.207519935 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71508608775168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14762196 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76221979297792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1949 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74856510846976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19724993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4988582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74856510846976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19714837 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74856527886336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1365451411456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1365363382784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34487189 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34477031 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34487189 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13548719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13548471 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1405340192768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7844976882964895e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34477031œžòçy0ú+ _wandbruntime‚762  _runtime‚762Òs¸€ûü _step‚157 /loss/FRt2/assignment_loss‚1.978413165740247 1loss/FBt2/assignment_loss‚0.16654548996506788 /loss/FRt1/detection_loss‚0.5475533718199065 .loss/FBt2/detection_loss‚2.496105360031336 % _timestamp‚1.7844976882974012e+09 1loss/SRqqt1/assignment_loss‚1.468311662497246 1loss/SRqqt2/detection_loss‚0.8363275295803317 1loss/SRqqt1/detectionYbA‚_loss‚2.4308062957786247 epoch‚2 trainer/global_step‚1949 _runtime‚ 762.207997626 0loss/FRt1/assignment_loss‚3.4775691288467807 2loss/SRqqt2/assignment_loss‚1.0492582763421645 0loss/FBt1/assignment_loss‚0.3497275416776856 /loss/FRt2/detection_loss‚1.3745849701675144 /loss/FBt1/detection_loss‚2.6343724842917675 %loss/total_loss‚18.80957527673867Üåó&ü /loss/FRt1/detection_loss‚0.5475533718199065 .loss/FBt2/detection_loss‚2.496105360031336 % _timestamp‚1.7844976882974012e+09 1loss/SRqqt1/assignment_loss‚1.468311662497246 1loss/SRqqt2/detection_loss‚0.8363275295803317 1loss/SRqqt1/detection_loss‚2.4308062957786247 epoch‚2 trainer/global_step‚1949 _runtime‚ 762.207997626 0loss/FRt1/assignment_loss‚3.4775691288467807 2loss/SRqqt2/assignment_loss‚1.0492582763421645 0loss/FBt1/assignment_loss‚0.3497275416776856 /loss/FRt2/detection_loss‚1.3745849701675144 /loss/FBt1/detection_loss‚2.6343724842917675 %loss/total_loss‚18.80957527673867 _step‚157 /loss/FRt2/assignment_loss‚1.978413165740247 1loss/FBt2/assignment_loss‚0.16654548996506788¡›S;ý‚(:° ›ŒõÒà¡Ï+ network.sent‚ 2.379704e+06 network.recv‚267068 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2254.58251953125 disk.md0.out‚ 27.0859375* proc.memory.percent‚17.218542098999023) disk./.usagePercent‚75.92549388483592% disk./.usageGB‚1165.6461448669434 disk.dm-1.out‚268.1396484375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚2378.6982421875$ memory_percent‚99.20637607574463( proc.memory.availableMB‚ 130.02734375 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2254.58251953125 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-0.out‚4307.81298828125" proc.memory.rssMB‚ 2821.0859375 cpu‚26.08432374570744 proc.cpu.threads‚17 disk.dm-1.in‚0$ disk.nvme1n1.out‚2378.6982421875 disk.dm-0.in‚0…g–8þ‚(:‚ ›ŒõÒ¥¸û+ gpu.0.gpu‚59.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.57' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.19 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­€Ä$8ÿ‚(:­ ªŒõÒ•Ç®+$ disk./.usageGB‚1165.646354675293" disk.dm-0.out‚4308.92333984375' disk.nvme0c0n1.out‚2255.91845703125% disk.nvme0n1.out‚2255.91845703125" proc.memory.rssMB‚ 2821.0859375* proc.memory.percent‚17.218542098999023 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚268.2490234375 disk.rbd0.out‚0$ memory_percent‚99.20639991760254 cpu‚26.082116222935486 network.recv‚271258% disk.nvme1c1n1.out‚2379.103515625 disk.md0.out‚ 27.3203125 disk.dm-0.in‚0 disk.nvme0n1.in‚0) disk./.usagePercent‚75.92550755090656# disk.nvme1n1.out‚2379.103515625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0' proc.memory.availableMB‚ 130.0234375 network.sent‚ 2.429078e+06„È%›€ ‚(: ªŒõÒÃæ+ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚197.741' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.91366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W}¶º0 + _wandbruntime‚780  _runtime‚780¦·~È‚  ' lr-AdamW/pg1‚0.00010152368993063405 ' lr-AdamW/pg2‚0.00010152368993063405 trainer/global_step‚1999 % _timestamp‚1.7844977066878386e+09 _runtime‚ 780.600378024 _step‚158‹xí‘̓ Ç _runtime‚ 780.600378024 _step‚158 ' lr-AdamW/pg1‚0.00010152368993063405 ' lr-AdamW/pg2‚0.00010152368993063405 trainer/global_step‚1999 % _timestamp‚1.7844977066878386e+09žç]Z­0„ + _wandbruntime‚780  _runtime‚780öà磿'… ¹O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15120925 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1398623014912 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78086085856256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15120925 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439577268224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77084601267821 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15120923 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76687462841344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20211833 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75690377974416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76687479880704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35332758 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5109868 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20211833 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73257696384000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75690360935056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚1999 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1398711036416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77084704455651 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35322614 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1394240332765 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78086190917120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1398711036416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73257671480320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844977066884916e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20201691 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74697273652224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78086085856256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13879970 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1394326481235 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5109626 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439390562816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1398623014912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13880215 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76687479880704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76687462841344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ß;)UÄ'† ¾O ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1398623014912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13880215 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76687479880704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76687462841344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15120925 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1398623014912 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78086085856256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15120925 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439577268224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77084601267821 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15120923 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76687462841344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20211833 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75690377974416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76687479880704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35332758 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5109868 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20211833 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73257696384000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75690360935056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚1999 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1398711036416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77084704455651 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35322614 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1394240332765 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78086190917120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1398711036416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73257671480320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844977066884916e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20201691 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74697273652224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78086085856256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13879970 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1394326481235 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5109626 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439390562816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830ŸH@Ñ0‡ + _wandbruntime‚780  _runtime‚780žê }Òˆ ÃM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15128246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8774634 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20211589 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74733452053504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20211589 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73293366637568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20221745 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78124130333184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35339833 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35349991 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1399301124608 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35349991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 trainer/global_step‚1999 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77122149287025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15128244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77122252474967 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78124235401216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1394916284209 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78124235401216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76724846247936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13886684 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73293391541248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15128246 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15128244 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13886938 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74733669946880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1399301124608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440085415936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75727233002816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440278405632 _step‚160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76724846247936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35349991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 trainer/global_step‚1999 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77122149287025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15128244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77122252474967 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78124235401216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1394916284209 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78124235401216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76724846247936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13886684 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73293391541248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15128246 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15128244 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13886938 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74733669946880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1399301124608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440085415936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75727233002816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440278405632 _step‚160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76724846247936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8774634 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20211589 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74733452053504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20211589 R7ü0Š + _wandbruntime‚780  _runtime‚780†Í´‹ ý 0loss/FRt2/assignment_loss‚2.0755632753282507 2loss/SRqqt1/assignment_loss‚1.4468192727970226 2loss/SRqqt2/assignment_loss‚0.9562062979205465 1loss/FBt2/assignment_loss‚0.11226561482261685 /loss/SRqqt2/detection_loss‚0.83354216136429 /loss/FBt1/detection_loss‚1.5288957038398496 &loss/total_loss‚16.513935748003263 _runtime‚ 780.855235208 1loss/FBt1/assignment_loss‚0.20611556420417856 /loss/FBt2/detection_loss‚2.1833806753679204 epoch‚2 % _timestamp‚1.7844977069446304e+09 0loss/FRt1/assignment_loss‚3.3891940373798897 /loss/FRt1/detection_loss‚0.6503310452201467 .loss/FRt2/detection_loss‚1.409408524814923 0loss/SRqqt1/detection_loss‚1.722213574943627 trainer/global_step‚1999 _step‚161¨Æi9Œ ‚ 0loss/FRt1/assignment_loss‚3.3891940373798897 /loss/FRt1/detection_loss‚0.6503310452201467 .loss/FRt2/detection_loss‚1.409408524814923 0loss/SRqqt1/detection_loss‚1.722213574943627 trainer/global_step‚1999 _step‚161 0loss/FRt2/assignment_loss‚2.0755632753282507 2loss/SRqqt1/assignment_loss‚1.4468192727970226 2loss/SRqqt2/assignment_loss‚0.9562062979205465 1loss/FBt2/assignment_loss‚0.11226561482261685 /loss/SRqqt2/detection_loss‚0.83354216136429 /loss/FBt1/detection_loss‚1.5288957038398496 &loss/total_loss‚16.513935748003263 _runtime‚ 780.855235208 1loss/FBt1/assignment_loss‚0.20611556420417856 /loss/FBt2/detection_loss‚2.1833806753679204 epoch‚2 % _timestamp‚1.7844977069446304e+09¡¨&3 ‚(:¨ ¹ŒõÒƒúÖ+) disk./.usagePercent‚75.95646368566233% disk./.usageGB‚1166.1216087341309$ disk.nvme0n1.out‚3157.8271484375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 5382.93359375 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚3157.8271484375 network.sent‚ 2.447715e+06 network.recv‚275041 disk.nvme0n1.in‚0% disk.nvme1c1n1.out‚2552.099609375 disk.md0.out‚ 27.6796875$ memory_percent‚99.20668601989746' proc.memory.availableMB‚ 129.9765625" proc.memory.rssMB‚ 2821.0859375# disk.nvme1n1.out‚2552.099609375 disk.dm-1.out‚268.3623046875 disk.rbd0.in‚0* proc.memory.percent‚17.218542098999023 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚26.07871429883602 proc.cpu.threads‚17áŽí¯šŽ ‚(: ¹ŒõÒ¤ª€, gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚148.87' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚49.623333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ800 + _wandbruntime‚799  _runtime‚799 ÁáÈ Â ' lr-AdamW/pg1‚0.00010150378865119817 ' lr-AdamW/pg2‚0.00010150378865119817 trainer/global_step‚2049 % _timestamp‚1.7844977251129644e+09 _runtime‚ 799.023923011 _step‚162ä6Í‘ Ç ' lr-AdamW/pg1‚0.00010150378865119817 ' lr-AdamW/pg2‚0.00010150378865119817 trainer/global_step‚2049 % _timestamp‚1.7844977251129644e+09 _runtime‚ 799.023923011 _step‚162¢“æûû0’ + _wandbruntime‚799  _runtime‚799½;{{¿'“ ¹O VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20698443 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75042429342720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1428155956643 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1432648778240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78962034131229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5233517 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1432560756736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78555781202944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78555798242304 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76516965602304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79988341959680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78555781202944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1474330941952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79988447020544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15486973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78555798242304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75042454246400 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8984847 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77533981362416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20698443 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8984844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79988341959680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20708585 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 _step‚163 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36195560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413952 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78962137319059 % _timestamp‚1.7844977251138105e+09 trainer/global_step‚2049 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1428069808173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20708585 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15486975 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15486973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77533964323056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1474511355904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1432648778240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15486975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5233273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚ 799.028384808«$” ¾O IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚ 799.028384808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20698443 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75042429342720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1428155956643 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1432648778240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78962034131229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5233517 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1432560756736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78555781202944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78555798242304 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76516965602304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79988341959680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78555781202944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1474330941952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79988447020544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15486973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78555798242304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75042454246400 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8984847 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77533981362416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20698443 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8984844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79988341959680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20708585 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 60vk䆠 94181451 _step‚163 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36195560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413952 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78962137319059 % _timestamp‚1.7844977251138105e+09 trainer/global_step‚2049 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1428069808173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20708585 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15486975 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15486973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77533964323056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1474511355904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1432648778240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15486975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5233273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 I0• + _wandbruntime‚799  _runtime‚799JçŲÈ&– ÂM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433238866432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20708341 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78999582150433 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80026491504640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75078124499968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚164 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80026386436608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15494296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77570836390816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15494294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77570853430176 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78593147570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1475031128064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78593147570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36202635 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1433326895104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36212793 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15494296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8989134 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚2049 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76553155628032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428745759617 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78593164609536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14225107 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20718497 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 $ _timestamp‚1.784497725371552e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433238866432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80026491504640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15494294 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5235741 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36212793 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20708341 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20718497 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176212480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14224875 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8989137 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78593164609536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1475207340544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75078149403648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75078149403648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78999582150433 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80026491504640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75078124499968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚164 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80026386436608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15494296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77570836390816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15494294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77570853430176 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78593147570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1475031128064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78593147570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36202635 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1433326895104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36212793 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15494296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8989134 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚2049 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76553155628032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428745759617 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78593164609536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14225107 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20718497 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 $ _timestamp‚1.784497725371552e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433238866432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80026491504640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15494294 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5235741 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36212793 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20708341 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20718497 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176212480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14224875 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8989137 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78593164609536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1475207340544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14555994 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15853023 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80424099564544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81890703123968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78336668427264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1509269613056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199347 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80424116603904 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81890703123968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76827212108800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76827187205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81890598063104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482Ä4ÿqÂ'¢ ¼O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15853025 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79377567711056 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21195195 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚ 817.838334091 _step‚167 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21205337 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80839466994637 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79377584750416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81890598063104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37058362 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5356901 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14555994 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15853023 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80424099564544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81890703123968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78336668427264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1509269613056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199347 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80424116603904 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81890703123968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76827212108800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76827187205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81928747608064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80461465931776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7844977441818311e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21215249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5359112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1462661383607 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15860346 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9203634 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78373058651648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37065437 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1467176608256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1510151385600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚2099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21215249 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78372859632640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80877015013841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80461482971136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81928642540032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76862882362368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81928747608064 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80877118201783 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1509977270272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1462575235025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80461482971136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15860346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37075595 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14563007 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15860344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9203637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1467264636928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21205093 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14562746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79414456818176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1467176608256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80461465931776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5359370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76862907266048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15860344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360õ7â´#¥ ÈM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37075595 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21205093 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81928747608064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80461465931776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7844977441818311e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21215249 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5359112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1462661383607 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15860346 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9203634 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78373058651648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37065437 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1467176608256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1510151385600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚2099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21215249 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78372859632640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80877015013841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80461482971136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81928642540032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76862882362368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81928747608064 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80877118201783 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1509977270272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1462575235025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80461482971136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15860346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37075595 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14563007 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15860344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9203637 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1467264636928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21205093 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14562746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79414456818176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1467176608256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80461465931776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5359370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76862907˜| 266048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15860344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _runtime‚ 818.092919415 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¨®T0¦ + _wandbruntime‚818  _runtime‚8188|ƒE§ € epoch‚2 0loss/FRt1/assignment_loss‚3.4135847189105473 0loss/FRt2/assignment_loss‚1.9703666035002068 2loss/SRqqt1/assignment_loss‚1.1022520224159618 1loss/FBt2/assignment_loss‚0.20677728422212696 1loss/SRqqt1/detection_loss‚2.7630561380559278 _runtime‚ 818.093811556 1loss/FBt1/assignment_loss‚0.28736227267519937 /loss/FRt2/detection_loss‚1.4635042197825656 &loss/total_loss‚18.330210795294576 trainer/global_step‚2099 2loss/SRqqt2/assignment_loss‚0.9531135760092487 1loss/SRqqt2/detection_loss‚0.8012496408825456 /loss/FBt1/detection_loss‚2.0597068145378636 % _timestamp‚1.7844977441827838e+09 _step‚169 /loss/FRt1/detection_loss‚0.5051101066188634 .loss/FBt2/detection_loss‚2.804127397683517`Xãû ¨ … &loss/total_loss‚18.330210795294576 trainer/global_step‚2099 2loss/SRqqt2/assignment_loss‚0.9531135760092487 1loss/SRqqt2/detection_loss‚0.8012496408825456 /loss/FBt1/detection_loss‚2.0597068145378636 % _timestamp‚1.7844977441827838e+09 _step‚169 /loss/FRt1/detection_loss‚0.5051101066188634 .loss/FBt2/detection_loss‚2.804127397683517 epoch‚2 0loss/FRt1/assignment_loss‚3.4135847189105473 0loss/FRt2/assignment_loss‚1.9703666035002068 2loss/SRqqt1/assignment_loss‚1.1022520224159618 1loss/FBt2/assignment_loss‚0.20677728422212696 1loss/SRqqt1/detection_loss‚2.7630561380559278 _runtime‚ 818.093811556 1loss/FBt1/assignment_loss‚0.28736227267519937 /loss/FRt2/detection_loss‚1.4635042197825656©ËÔàª2© ‚(:§ ׌õÒÒê+ proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 network.sent‚ 2.571242e+06% disk./.usageGB‚1166.0754661560059 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚3422.572265625 disk.md0.in‚0! disk.dm-0.out‚5695.3056640625 disk.rbd0.in‚0# disk.nvme0n1.out‚3422.572265625& disk.nvme1c1n1.out‚2601.8564453125$ memory_percent‚99.60207939147949& proc.memory.availableMB‚ 65.1953125 network.recv‚283850) disk./.usagePercent‚75.95345814401601 disk.dm-1.out‚ 268.69140625 disk.rbd0.out‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚2601.8564453125 disk.dm-0.in‚0" proc.memory.rssMB‚ 2821.0859375 disk.dm-1.in‚0 disk.md0.out‚ 28.515625* proc.memory.percent‚17.218542098999023 cpu‚26.072903738005042®²÷2™ª ‚(:Ž ׌õÒ¥¥’, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.318' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.43933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›JU50« + _wandbruntime‚836  _runtime‚836V_XȬ  trainer/global_step‚2149 % _timestamp‚1.7844977626996608e+09 _runtime‚ 836.611150285 _step‚170 ' lr-AdamW/pg1‚0.00010146155383409429 ' lr-AdamW/pg2‚0.00010146155383409429Iä\|Í­ Ç _runtime‚ 836.611150285 _step‚170 ' lr-AdamW/pg1‚0.00010146155383409429 ' lr-AdamW/pg2‚0.00010146155383409429 trainer/global_step‚2149 % _timestamp‚1.7844977626996608e+09ª+j¹"0® + _wandbruntime‚836  _runtime‚836ÃBv¿'¯ ¹O % _timestamp‚1.7844977627002995e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82292434965504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480216 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82716899858045 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16219073 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82292434965504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83792854166528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500524261888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16219073 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1500436240384 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37911020 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21702089 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21691947 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1544265897472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1495728758989 trainer/global_step‚2149 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14894309 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78611969971200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1495814907459 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21702089 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82292417926144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9413847 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80156210964992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9413844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78611945067520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 836.611210635 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16219075 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1500436240384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80156420476928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104Åé‹Ä'° ¾O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78611945067520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 836.611210635 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16219075 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1500436240384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80156420476928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844977627002995e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82292434965504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480216 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82716899858045 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16219073 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82292434965504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83792854166528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500524261888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16219073 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1500436240384 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37911020 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21702089 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21691947 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1544265897472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1495728758989 trainer/global_step‚2149 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14894309 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78611969971200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1495814907459 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21702089 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82292417926144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9413847 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80156210964992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9413844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152«;0± + _wandbruntime‚836  _runtime‚836…r9Jв ÃM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚2149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501202378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21712001 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1496490859015 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚172 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37938397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14900811 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1496404710433 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14901065 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37928239 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚ 836.868095792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21701845 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81258060206176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9418137 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83831003711488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16226396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1545148718592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37938397 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82329784293376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21712001 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 VCDeviceStatsMonitor.on_train_batch_end/n° ùrequested_bytes.all.allocated‚82754551065191 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82329801332736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83830898643456 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16226394 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82754447877249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81258043166816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78647640224768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1544966214656 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582pͨJÎ&³ ÈM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚2149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501202378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21712001 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1496490859015 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚172 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37938397 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14900811 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1496404710433 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14901065 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37928239 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚ 836.868095792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21701845 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81258060206176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9418137 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83831003711488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16226396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1545148718592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37938397 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82329784293376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21712001 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82754551065191 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82329801332736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83830898643456 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16226394 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82754447877249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81258043166816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78647640224768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1544966214656 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330¬^‚D,0´ + _wandbruntime‚836  _runtime‚836xÝ’µ þ 1loss/FBt1/assignment_loss‚0.12631411680011298 /loss/FBt2/detection_loss‚1.5577079658899038 trainer/global_step‚2149 2loss/SRqqt2/assignment_loss‚1.0825785307758768 0loss/FBt2/assignment_loss‚0.0359199056926122 /loss/FRt1/detection_loss‚0.5769526949501597 /loss/FRt2/detection_loss‚1.3526465505201708 % _timestamp‚1.7844977629579442e+09 _runtime‚ 836.868588223 _step‚173 /loss/FRt1/assignment_loss‚3.258812592647694 0loss/FRt2/assignment_loss‚1.8087085321408893 1loss/SRqqt1/assignment_loss‚1.264617992968609 1loss/SRqqt1/detection_loss‚2.3351187244710068 1loss/SRqqt2/detection_loss‚0.7142011311662908 /loss/FBt1/detection_loss‚2.2997060341653377 &loss/total_loss‚16.413284772188664 epoch‚2³(RM ¶ ƒ /loss/FBt1/detection_loss‚2.2997060341653377 &loss/total_loss‚16.413284772188664 epoch‚2 1loss/FBt1/assignment_loss‚0.12631411680011298 /loss/FBt2/detection_loss‚1.5577079658899038 trainer/global_step‚2149 2loss/SRqqt2/assignment_loss‚1.0825785307758768 0loss/FBt2/assignment_loss‚0.0359199056926122 /loss/FRt1/detection_loss‚0.5769526949501597 /loss/FRt2/detection_loss‚1.3526465505201708 % _timestamp‚1.7844977629579442e+09 _runtime‚ 836.868588223 _step‚173 /loss/FRt1/assignment_loss‚3.258812592647694 0loss/FRt2/assignment_loss‚1.8087085321408893 1loss/SRqqt1/assignment_loss‚1.264617992968609 1loss/SRqqt1/detection_loss‚2.3351187244710068 1loss/SRqqt2/detection_loss‚0.7142011311662908­Ù‰ 9· ‚(:® æŒõÒÂÐâ+ disk.md0.in‚0" proc.memory.rssMB‚ 2821.0859375 cpu‚26.071039804534827) disk./.usagePercent‚75.93796951649263' disk.nvme0c0n1.out‚3423.38232421875 disk.nvme0n1.in‚0! disk.dm-0.out‚5695.8173828125 disk.dm-1.out‚ 268.75390625* proc.memory.percent‚17.218542098999023% disk./.usageGB‚1165.8376770019531 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 28.8671875 disk.dm-1.in‚0& proc.memory.availableMB‚ 65.1484375 network.recv‚288097 disk.nvme0c0n1.in‚0% disk.nvme1n1.out‚2602.36572265625 disk.dm-0.in‚0$ memory_percent‚99.60236549377441 proc.cpu.threads‚17 network.sent‚ 2.633458e+06% disk.nvme0n1.out‚3423.38232421875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2602.36572265625 disk.nvme1n1.in‚0f̸ ‚(:„ æŒõÒꪉ, gpu.0.gpu‚69.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚128.229' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚42.743 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Ÿ¢;8¹ ‚(:­ õŒõÒ‹þ·+! disk.dm-1.out‚268.95556640625 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚3425.4326171875 disk.nvme0n1.in‚0% proc.memory.availableMB‚ 64.890625 cpu‚26.080200772906675 network.sent‚ 2.684633e+06 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚2602.96533203125 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚3425.4326171875 proc.cpu.threads‚17( disk./.usagePercent‚75.9379265304886% disk./.usageGB‚1165.8370170593262! disk.dm-0.out‚5697.5947265625 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.60393905639648 network.recv‚292749' disk.nvme1c1n1.out‚2602.96533203125" proc.memory.rssMB‚ 2821.0859375* proc.memory.percent‚17.218542098999023 disk.nvme1n1.in‚0 disk.md0.out‚ 29.1796875Q[vm›º ‚(: õŒõÒ´°à+ gpu.0.gpu‚57.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚121.999' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.666333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÙ Æ0» + _wandbruntime‚856  _runtime‚856ãõc‘Ǽ Á ' lr-AdamW/pg1‚0.00010143922097290262 ' lr-AdamW/pg2‚0.00010143922097290262 trainer/global_step‚2199 $ _timestamp‚1.784497782245757e+09 _runtime‚ 856.156698296 _step‚174™[7Y̽ Æ _runtime‚ 856.156698296 _step‚174 ' lr-AdamW/pg1‚0.00010143922097290262 ' lr-AdamW/pg2‚0.00010143922097290262 trainer/global_step‚2199 $ _timestamp‚1.784497782245757e+09®mùåÑ0¾ + _wandbruntime‚856  _runtime‚856ò¨M›¾'¿ ¸O G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38773822 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85695110269952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1579166289408 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22188699 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 856.157584328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1534373982208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15231723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80396702929920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1529644382867 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22198841 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84160753327104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84160753327104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 $ _timestamp‚1.784497782246692e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5603379 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85695215330816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1534373982208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1579350897664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85695215330816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9628347 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9628344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83064791526416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81975869219328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15231968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22188699 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84160736287744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16585123 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16585123 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81976078731264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83064774487056 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84160736287744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85695110269952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38783966 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5603621 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22188699 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 856.157584328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1534373982208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15231723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80396702929920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1529644382867 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22198841 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84160753327104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84160753327104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 $ _timestamp‚1.784497782246692e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5603379 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85695215330816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1534373982208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1579350897664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85695215330816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9628347 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9628344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83064791526416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81975869219328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15231968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22188699 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84160736287744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16585123 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16585123 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81976078731264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83064774487¢äl|056 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84160736287744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85695110269952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38783966 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5603621 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38801199 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82012473018880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22208753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15238535 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83101663594176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚ 856.429127967 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22198597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9632634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 trainer/global_step‚2199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82012263514112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84198102654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16592446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84198102654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1535140120576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1535052091904 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22198597 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16592446 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5605901 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84631983928599 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84631880740657 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5606161 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1579865426944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83101646554816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15238798 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85733259814912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184601088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16592444 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84198119694336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1530320334423 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16592444 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38791041 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1535140120576¥Y…¹Î&à ÈM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84198119694336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1530320334423 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16592444 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38791041 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1535140120576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844977825180392e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38801199 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82012473018880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22208753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15238535 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83101663594176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚ 856.429127967 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22198597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9632634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 trainer/global_step‚2199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82012263514112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84198102654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16592446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84198102654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1535140120576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1535052091904 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22198597 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16592446 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5605901 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84631983928599 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84631880740657 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5606161 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1579865426944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83101646554816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15238798 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85733259814912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184601088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16592444 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86029054649344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15569770 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16951175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15569519 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82181460792320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1614115113984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84908394914416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚2249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39646768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22695593 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚ 876.377416699 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16951175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86029054649344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86029071688704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚87597471434240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87597366373376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1568399745536 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86471868772691 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86471765584861 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1563473858275 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16951175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86029054649344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86029071688704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚87597471434240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87597366373376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1568399745536 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86471868772691 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86471765584861 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1563473858275 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86029054649344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15569770 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16951175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15569519 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82181460792320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1614115113984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84908394914416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚2249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39646768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482³m¹œ§0Ï + _wandbruntime‚876  _runtime‚876bÚ%É&Ð ÃM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22695349 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39653843 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87635515918336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1564149809831 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86509416792007 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15576226 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83832182523904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22705505 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16958496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16958496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 trainer/global_step‚2249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16958494 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1569077862400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1569077862400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84945266982176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16958494 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86066421016576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1568989833728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 876.641089695 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84945249942816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39664001 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39664001 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87635515918336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695349 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86509313604065 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1564063661249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9847134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82217180853248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1614812875264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87635410850304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1568989833728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705505 % _timestamp‚1.7844978027300353e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86066421016576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82217155949568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9847137 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5729092 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15576226 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83832182523904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22705505 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16958496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16958496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 trainer/global_step‚2249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16958494 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1569077862400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1569077862400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84945266982176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16958494 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86066421016576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1568989833728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 876.641089695 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84945249942816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39664001 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39664001 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87635515918336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695349 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86509313604065 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1564063661249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.larg¶WÃ.e_pool.freed‚9847134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82217180853248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1614812875264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87635410850304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1568989833728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705505 % _timestamp‚1.7844978027300353e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86066421016576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82217155949568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9847137 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5729092 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 BÕ ‚(:³ “õÒ¸Žá+' proc.memory.availableMB‚ 65.04296875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚2645.40771484375 disk.md0.in‚0 disk.md0.out‚ 30.08984375$ memory_percent‚99.60300922393799" proc.memory.rssMB‚ 2821.0859375 cpu‚26.116194946038302' disk.nvme0c0n1.out‚3531.13037109375) disk./.usagePercent‚75.95341615190803 disk.dm-0.in‚0" disk.dm-0.out‚5843.62548828125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚300081$ disk./.usageGB‚1166.074821472168 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚2645.40771484375* proc.memory.percent‚17.218542098999023% disk.nvme0n1.out‚3531.13037109375 network.sent‚ 2.762004e+06! disk.dm-1.out‚269.14501953125 disk.nvme1n1.in‚0=1Ç™Ö ‚(:Ž “õÒº¤Š, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.754' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.91799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;!èü0× + _wandbruntime‚896  _runtime‚896½%ÚÈØ Â % _timestamp‚1.7844978224354303e+09 _runtime‚ 896.346398973 _step‚182 ' lr-AdamW/pg1‚0.00010139212613376453 ' lr-AdamW/pg2‚0.00010139212613376453 trainer/global_step‚2299Ò‘­ÍÙ Ç ' lr-AdamW/pg2‚0.00010139212613376453 trainer/global_step‚2299 % _timestamp‚1.7844978224354303e+09 _runtime‚ 896.346398973 _step‚182 ' lr-AdamW/pg1‚0.00010139212613376453¶!V.0Ú + _wandbruntime‚896  _runtime‚896¢@·‘Á'Û »O _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86751998302416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1602249465856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17317223 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85615220144128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚183 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87897390050304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5849864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5849602 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1602337487360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87897390050304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40509570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89499622476800 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15907211 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89499622476800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195094016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 % _timestamp‚1.7844978224364412e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1602337487360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597217185213 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23192345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1649196583424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86751981263056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88349198448269 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17317225 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15906946 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10057344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1602249465856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1649001489408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40499426 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23192345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10057347 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83966218654720 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2}M[cÆ'Ü ÀO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1602249465856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17317223 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85615220144128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚183 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87897390050304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5849864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5849602 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1602337487360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87897390050304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40509570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89499622476800 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15907211 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89499622476800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195094016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 % _timestamp‚1.7844978224364412e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1602337487360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597217185213 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23192345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1649196583424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86751981263056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88349198448269 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17317225 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15906946 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10057344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1602249465856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1649001489408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40499426 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23192345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10057347 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83966218654720 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86751998302416·[`0Ó0Ý + _wandbruntime‚896  _runtime‚896È[QÎÞ ÅM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17324546 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15913581 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89537772021760 _step‚184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40516645 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15913833 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40526803 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84001913811968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 _runtime‚ 896.614768601 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603015604224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1649703969280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88386849655415 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602927575552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚89537772021760 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1597979285239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86788870370176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚2299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87934756417536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40526803 MGDˆksùý eviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23202257 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89537666953728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87934739378176 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88386746467473 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86788853330816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17324546 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23192101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17324544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚100616347›Á´Ð&ß ÊM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87934739378176 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85651617781248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40526803 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84001913811968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 _runtime‚ 896.614768601 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603015604224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1649703969280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88386849655415 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602927575552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚89537772021760 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1597979285239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86788870370176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚2299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87934756417536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40526803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23202257 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89537666953728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87934739378176 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88386746467473 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86788853330816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17324546 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23192101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17324544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10061634 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17324546 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15913581 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89537772021760 _step‚184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40516645 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15913833 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 Eä ‚(:³ ¢õÒ¤õÄ- network.recv‚303927) disk./.usagePercent‚75.95341292174588% disk./.usageGB‚1166.0747718811035% disk.nvme1n1.out‚2646.82763671875' disk.nvme1c1n1.out‚2646.82763671875$ memory_percent‚99.60324764251709' proc.memory.availableMB‚ 65.00390625" proc.memory.rssMB‚ 2821.0859375 network.sent‚ 2.811286e+06 disk.nvme1n1.in‚0" disk.dm-0.out‚5959.29443359375! disk.dm-1.out‚269.23876953125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚3646.46923828125% disk.nvme0n1.out‚3646.46923828125 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0* proc.memory.percent‚17.218542098999023 cpu‚26.12047650973327 disk.md0.in‚0 disk.md0.out‚ 30.55078125 disk.rbd0.out‚0 proc.cpu.threads‚17ç¾V%;å ‚(:° ±õÒ‹ï·+ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚2647.7333984375 network.recv‚308476 disk.md0.in‚0 disk.dm-0.in‚0! disk.dm-1.out‚269.49365234375& disk.nvme0c0n1.out‚3647.7138671875$ memory_percent‚99.60405826568604* proc.memory.percent‚17.218542098999023% disk./.usageGB‚1166.0748977661133 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚3647.7138671875$ disk.nvme1n1.out‚2647.7333984375 proc.cpu.threads‚17) disk./.usagePercent‚75.95342112138826 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 64.87109375" proc.memory.rssMB‚ 2821.0859375 cpu‚26.116767909740535 network.sent‚ 2.875482e+06 disk.md0.out‚ 30.92578125" disk.dm-0.out‚5960.37255859375 disk.dm-1.in‚0Ón0šæ ‚(: ±õÒ Ðæ+ gpu.0.gpu‚57.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚197.869' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.95633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Ãn0ç + _wandbruntime‚915  _runtime‚915^Ø‹Èè  _step‚186 ' lr-AdamW/pg1‚0.00010136736491013757 ' lr-AdamW/pg2‚0.00010136736491013757 trainer/global_step‚2349 % _timestamp‚1.7844978420609283e+09 _runtime‚ 915.971866486 %€Íé Ç _step‚186 ' lr-AdamW/pg1‚0.00010136736491013757 ' lr-AdamW/pg2‚0.00010136736491013757 trainer/global_step‚2349 % _timestamp‚1.7844978420609283e+09 _runtime‚ 915.971866486ºÏ„0ê + _wandbruntime‚915  _runtime‚9156õÁ'ë »O TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91401878580224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85750976517120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91401983641088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1636187207680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16245952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1636275229184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10271844 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87435163374080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17683275 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85751001420800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17683273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23678955 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636187207680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1631132809091 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17683275 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚88595584651056 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89765691372544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16245692 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5974105 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23689097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87434953862144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1683977345024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1684161953280 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90226631311677 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88595601690416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91401878580224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89765708411904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1636275229184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89765708411904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 915.972748138 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89765691372544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17683273 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10271847 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41372372 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23689097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90226734499507 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2349 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91401983641088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23678955 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85750976517120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91401983641088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚187 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1636187207680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16245952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1636275229184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10271844 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87435163374080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17683275 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85751001420800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17683273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23678955 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636187207680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1631132809091 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17683275 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚88595584651056 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89765691372544 TGDeviceStatsM.wýzœonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16245692 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5974105 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23689097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87434953862144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1683977345024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1684161953280 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90226631311677 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88595601690416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91401878580224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89765708411904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1636275229184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89765708411904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 915.972748138 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89765691372544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17683273 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10271847 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41372372 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23689097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90226734499507 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2349 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91401983641088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23678955 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41389605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85786671674368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91440028125184 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90264179330881 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1636953346048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89803074779136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89803057739776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16252650 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88632456718816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17690594 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23699009 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10276137 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90264282518823 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23699009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2349 $ _timestamp‚1.784497842328928e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89803057739776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87471559365632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684676089344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89803074779136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23688853 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91440028125184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41379447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1631808760647 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10276134 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17690596 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88632473758176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41389605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16252397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684862787584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87471347763712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23688853 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1631722612065 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41379447 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1636953346048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5976513 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808SÔµÏ&ï ÉM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89803074779136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23688853 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91440028125184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41379447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1631808760647 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10276134 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17690596 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88632473758176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41389605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16252397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684862787584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87471347763712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23688853 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1631722612065 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41379447 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1636953346048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5976513 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1636865317376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41389605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85786671674368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91440028125184 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90264179330881 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1636953346048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89803074779136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89803057739776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16252650 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88632456718816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17690594 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23699009 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10276137 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90264282518823 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23699009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2349 $ _timestamp‚1.784497842328928e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89803057739776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87471559365632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684676089344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¼xð§°0ð + _wandbruntime‚916  _runtime‚916~Dr¥ñ ü trainer/global_step‚2349 1loss/SRqqt1/detection_loss‚1.8723041840461803 _step‚189 0loss/FRt1/assignment_loss‚3.2418812335836398 1loss/SRqqt2/assignment_loss‚1.021640968016225 0loss/FBt1/assignment_loss‚0.2950359676793113 .loss/FBt2/detection_loss‚2.019819735678578 epoch‚2 % _timestamp‚1.7844978423305874e+09 /loss/FRt2/assignment_loss‚1.916597285971474 1loss/SRqqt1/assignment_loss‚1.310679169756568 /loss/FRt2/detection_loss‚1.4725172528089545 1loss/SRqqt2/detection_loss‚0.8657002215782443 /loss/FBt1/detection_loss‚1.7241976779120278 _runtime‚ 916.241196678 1loss/FBt2/assignment_loss‚0.19044635001701798 /loss/FRt1/detection_loss‚0.5770364656293067 &loss/total_loss‚16.507856512677527jõI‡ò  0loss/FRt1/assignment_loss‚3.2418812335836398 1loss/SRqqt2/assignment_loss‚1.021640968016225 0loss/FBt1/assignment_loss‚0.2950359676793113 .loss/FBt2/detection_loss‚2.019819735678578 epoch‚2 % _timestamp‚1.7844978423305874e+09 /loss/FRt2/assignment_loss‚1.916597285971474 1loss/SRqqt1/assignment_loss‚1.310679169756568 /loss/FRt2/detection_loss‚1.4725172528089545 1loss/SRqqt2/detection_loss‚0.8657002215782443 /loss/FBt1/detection_loss‚1.7241976779120278 _runtime‚ 916.241196678 1loss/FBt2/assignment_loss‚0.19044635001701798 /loss/FRt1/detection_loss‚0.5770364656293067 &loss/total_loss‚16.507856512677527 trainer/global_step‚2349 1loss/SRqqt1/detection_loss‚1.8723041840461803 _step‚189½*H' 9ó ‚(:® ÀõÒ…¯û+% disk.nvme1n1.out‚2650.05029296875$ memory_percent‚99.60432052612305* proc.memory.percent‚17.218542098999023) disk./.usagePercent‚75.95341863664814 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 31.17578125 cpu‚26.11434983543402 disk.dm-0.in‚0! disk.dm-1.out‚269.56787109375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚3656.0205078125' disk.nvme1c1n1.out‚2650.05029296875 disk.md0.in‚0% proc.memory.availableMB‚ 64.828125$ disk.nvme0n1.out‚3656.0205078125 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 2821.0859375 proc.cpu.threads‚17 network.sent‚ 2.891248e+06 network.recv‚311872% disk./.usageGB‚1166.0748596191406! disk.dm-0.out‚5970.3740234375 disk.nvme0n1.in‚0ª§ßæšô ‚(: ÀõÒüÿ«, gpu.0.gpu‚60.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚113.861' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.95366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õìe0õ + _wandbruntime‚935  _runtime‚935<-¡šÇö Á trainer/global_step‚2399 % _timestamp‚1.7844978616674824e+09 _runtime‚ 935.57845392 _step‚190 ' lr-AdamW/pg1‚0.00010134179500828734 ' lr-AdamW/pg2‚0.00010134179500828734Õ¦g¶Ì÷ Æ ' lr-AdamW/pg2‚0.00010134179500828734 trainer/global_step‚2399 % _timestamp‚1.7844978616674824e+09 _runtime‚ 935.57845392 _step‚190 ' lr-AdamW/pg1‚0.00010134179500828734¾Ïd*0ø + _wandbruntime‚935  _runtime‚935GØ:ù ºO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10486344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89254639119360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89254852825600 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92104064175085 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6097309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87535734379520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1670124949504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16583911 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚2399 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93304134683648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18049325 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87535759283200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24185849 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24175707 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6097564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1718904739840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93304239744512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91634026773504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24175707 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90439205078416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1719093542400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93304239744512ûÇñÅ'ú ¿O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90439188039056 _step‚191 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 935.579531702 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚91634026773504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1670124949504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18049325 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18049325 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87535759283200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24185849 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24175707 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6097564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1718904739840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93304239744512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91634026773504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24175707 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90439205078416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1719093542400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93304239744512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10486344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89254639119360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89254852825600 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92104064175085 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6097309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87535734379520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1670124949504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16583911 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚2399 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93304134683648 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87571429536768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24195761 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89291035268608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90476077146176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670803059200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10490637 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91671376101376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6100023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92141612194289 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93342179160576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚ 935.846533947 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92141715382231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16590660 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1665552087473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42242249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18056644 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16590419 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6099785 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90476060106816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844978619354424e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24195761 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670891087872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18056646 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42252407 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670803059200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42252407 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2399 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91671393140736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1719788235776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91671376101376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18056646 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3135656963F—ý ÊM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670891087872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18056646 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42252407 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670803059200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42252407 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2399 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91671393140736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1719788235776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91671376101376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18056646 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10490634 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1719605731840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87571429536768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24195761 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚48€83ÿ9 2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89291035268608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90476077146176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670803059200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10490637 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91671376101376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6100023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92141612194289 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93342179160576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚ 935.846533947 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92141715382231 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16590660 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1665552087473 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42242249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18056644 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16590419 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6099785 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90476060106816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844978619354424e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24195761ÀÐYM#0þ + _wandbruntime‚935  _runtime‚935ÑíÝ»ÿ ý /loss/FRt2/assignment_loss‚1.630554496117358 0loss/FBt1/assignment_loss‚0.2664875135065998 epoch‚2 2loss/SRqqt2/assignment_loss‚0.9111313833835956 1loss/FBt2/assignment_loss‚0.10326636226444166 /loss/FRt1/detection_loss‚0.5327242943955796 _runtime‚ 935.847133698 2loss/SRqqt1/assignment_loss‚1.2783013769433278 1loss/SRqqt1/detection_loss‚2.1208550764297054 &loss/total_loss‚14.796060009811931 trainer/global_step‚2399 /loss/FRt1/assignment_loss‚3.200214654774893 /loss/FRt2/detection_loss‚1.3570764913408924 0loss/SRqqt2/detection_loss‚0.755189820115943 /loss/FBt1/detection_loss‚1.2993869965027498 /loss/FBt2/detection_loss‚1.3408715440368457 % _timestamp‚1.7844978619364862e+09 _step‚193bþ¤€ ‚ 2loss/SRqqt2/assignment_loss‚0.9111313833835956 1loss/FBt2/assignment_loss‚0.10326636226444166 /loss/FRt1/detection_loss‚0.5327242943955796 _runtime‚ 935.847133698 2loss/SRqqt1/assignment_loss‚1.2783013769433278 1loss/SRqqt1/detection_loss‚2.1208550764297054 &loss/total_loss‚14.796060009811931 trainer/global_step‚2399 /loss/FRt1/assignment_loss‚3.200214654774893 /loss/FRt2/detection_loss‚1.3570764913408924 0loss/SRqqt2/detection_loss‚0.755189820115943 /loss/FBt1/detection_loss‚1.2993869965027498 /loss/FBt2/detection_loss‚1.3408715440368457 % _timestamp‚1.7844978619364862e+09 _step‚193 /loss/FRt2/assignment_loss‚1.630554496117358 0loss/FBt1/assignment_loss‚0.2664875135065998 epoch‚2Á~‚6 ‚(:« ÏõÒ¼ûÝ+" disk.nvme0n1.out‚ 3656.98828125# proc.memory.rssMB‚ 2823.06640625 cpu‚26.13762231720089( disk./.usagePercent‚75.9534241030764 disk.md0.in‚0 disk.nvme0c0n1.in‚0# memory_percent‚99.6171236038208 proc.cpu.threads‚17 disk.nvme1n1.in‚0! disk.dm-1.out‚269.74658203125 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2650.76708984375' proc.memory.availableMB‚ 62.73046875 network.sent‚ 2.953381e+06 network.recv‚316411 disk.md0.out‚ 31.62890625 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 3656.98828125* proc.memory.percent‚17.230629920959473% disk./.usageGB‚1166.0749435424805% disk.nvme1n1.out‚2650.76708984375! disk.dm-0.out‚5970.8974609375 disk.dm-1.in‚0 disk.nvme0n1.in‚0ບ‚ ‚(: ÏõÒÛƒŠ, gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.269' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.08966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ù ¨0ƒ + _wandbruntime‚952  _runtime‚952 `Ûþ=„ ¸{ 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16171711683273315 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2957748770713806 *jet/accuracy_1_of_2‚0.7322279214859009 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7055381536483765 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.716666579246521 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.938034176826477 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9701940417289734 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 *jet/accuracy_1_of_1‚0.2734113335609436 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7285087704658508 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9561403393745422 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1458333283662796 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5598745346069336 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3522283732891083 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5782409906387329 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.112230584025383 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19204244017601013 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.8908089399337769 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1539159119129181 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9875569939613342 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9932739734649658 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14602859318256378 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13039448857307434 jet/accuracy_4_of_5‚0 trainer/global_step‚2402 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20055636763572693 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975950121879578 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1359943151473999 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.968152642250061 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1342887580394745 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6241033673286438 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.25977620482444763 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33695369958877563 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17829057574272156 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5585705041885376 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4286320209503174 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1359943151473999 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004961125086992979 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15683121979236603 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7200000286102295 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_4_of_4‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9751313328742981 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9743407964706421 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14368762075901031 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009260655380785465 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21941259503364563 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9312669038772583 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12124933302402496 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6917919516563416 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9190823435783386 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9824514985084534 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.997619092464447 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.4049246609210968 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6830280423164368 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.17629782855510712 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48888537287712097 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.907943844795227 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6656010150909424 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.35550656914711 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04791666939854622 _step‚194 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11877259612083435 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030051227658987045 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2150435447692871 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17974956333637238 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5859659314155579 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14681938290596008 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03625812754034996 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16532477736473083 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.547945499420166 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3142654299736023 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5605853199958801 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8436182141304016 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6495383381843567 'particle/accuracy‚0.832062304019928 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8365615010261536 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6461575627326965 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1439586579799652 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10543559491634369 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1350381076335907 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6284217834472656 +particle/sensitivity‚0.5798323750495911 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5644736886024475 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3928571343421936 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2857142984867096 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9218092560768127 *loss/val_total_loss‚15.889991425116404 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11673568189144135 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9901794195175171 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1892116814851761 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.30000001192092896 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3928571343421936 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.509829044342041 jet/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6267942190170288 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9884598255157471 *jet/accuracy_1_of_3‚0.8296731114387512 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.41906753182411194 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9901875257492065 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5803913474082947 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.47125911712646484 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5803913474082947 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7200000286102295 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15342627465724945 /particle/accuracy_3_of_3‚0.9984782934188843 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 +jet/accuracy_2_of_2‚0.10228663682937622 *jet/accuracy_2_of_4‚0.4982822835445404 /particle/accuracy_3_of_4‚0.9888125061988831 particle/accuracy_3_of_5‚1 _runtime‚ 952.680703306 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.994533360004425 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35185185074806213 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5656585097312927 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.615964949131012 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9925734996795654 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17089831829071045 particle/accuracy_2_of_4‚1 &particle/f_score‚0.597219705581665 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.30000001192092896 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8055555820465088 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9932739734649658 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1524272859096527 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6141022443771362 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.593393087387085 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4713653326034546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5656542181968689 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 epoch‚2 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2625221312046051 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9637782573699951 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.153524711728096 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4890916049480438 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.37334737181663513 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26499998569488525 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4890916049480438 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21755816042423248 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5876516699790955 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6443241238594055 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 % _timestamp‚1.7844978787692807e+09 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3239608108997345 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2667849659919739 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9726389050483704 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7971314787864685 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4409722089767456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4749833941459656 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7552043199539185 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16307446360588074 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7666666507720947 +jet/accuracy_3_of_3‚0.04575033858418465 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5788716077804565 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.142364501953125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6628066301345825 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25100961327552795 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47917094826698303 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3385416567325592 particle/accuracy_1_of_4‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03574349358677864 *particle/specificity‚0.901035487651825 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5292776226997375 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5480059385299683 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9978355169296265 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5118833780288696 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002914011711254716 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005104463547468185 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9663338661193848 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9876322746276855 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5611194372177124 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15342627465724945 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.8032132983207703 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4864671230316162 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5098010301589966 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6164886951446533 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7911236882209778 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34154707193374634f·/)… ½{ 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16532477736473083 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.547945499420166 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3142654299736023 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5605853199958801 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8436182141304016 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6495383381843567 'particle/accuracy‚0.832062304019928 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8365615010261536 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6461575627326965 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1439586579799652 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10543559491634369 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1350381076335907 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6284217834472656 +particle/sensitivity‚0.5798323750495911 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5644736886024475 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3928571343421936 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2857142984867096 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9218092560768127 *loss/val_total_loss‚15.889991425116404 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11673568189144135 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9901794195175171 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1892116814851761 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.30000001192092896 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3928571343421936 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.509829044342041 jet/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6267942190170288 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9884598255157471 *jet/accuracy_1_of_3‚0.8296731114387512 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.41906753182411194 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9901875257492065 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5803913474082947 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.47125911712646484 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5803913474082947 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7200000286102295 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15342627465724945 /particle/accuracy_3_of_3‚0.9984782934188843 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 +jet/accuracy_2_of_2‚0.10228663682937622 *jet/accuracy_2_of_4‚0.4982822835445404 /particle/accuracy_3_of_4‚0.9888125061988831 particle/accuracy_3_of_5‚1 _runtime‚ 952.680703306 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.994533360004425 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35185185074806213 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5656585097312927 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.615964949131012 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9925734996795654 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17089831829071045 particle/accuracy_2_of_4‚1 &particle/f_score‚0.597219705581665 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.30000001192092896 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8055555820465088 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9932739734649658 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1524272859096527 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6141022443771362 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.593393087387085 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4713653326034546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5656542181968689 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 epoch‚2 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2625221312046051 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9637782573699951 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.153524711728096 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4890916049480438 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.37334737181663513 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26499998569488525 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4890916049480438 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21755816042423248 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5876516699790955 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6443241238594055 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 % _timestamp‚1.7844978787692807e+09 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3239608108997345 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2667849659919739 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9726389050483704 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7971314787864685 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4409722089767456 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4749833941459656 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7552043199539185 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16307446360588074 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7666666507720947 +jet/accuracy_3_of_3‚0.04575033858418465 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5788716077804565 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.142364501953125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6628066301345825 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25100961327552795 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47917094826698303 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3385416567325592 particle/accuracy_1_of_4‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03574349358677864 *particle/specificity‚0.901035487651825 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5292776226997375 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5480059385299683 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9978355169296265 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5118833780288696 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002914011711254716 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005104463547468185 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9663338661193848 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9876322746276855 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5611194372177124 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15342627465724945 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.8032132983207703 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4864671230316162 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5098010301589966 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6164886951446533 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7911236882209778 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34154707193374634 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16171711683273315 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2957748770713806 *jet/accuracy_1_of_2‚0.7322279214859009 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7055381536483765 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.716666579246521 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.938034176826477 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9701940417289734 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 *jet/accuracy_1_of_1‚0.2734113335609436 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7285087704658508 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9561403393745422 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚07>…4.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1458333283662796 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5598745346069336 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3522283732891083 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5782409906387329 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.112230584025383 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19204244017601013 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.8908089399337769 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1539159119129181 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9875569939613342 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9932739734649658 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14602859318256378 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13039448857307434 jet/accuracy_4_of_5‚0 trainer/global_step‚2402 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20055636763572693 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975950121879578 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1359943151473999 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.968152642250061 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1342887580394745 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6241033673286438 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.25977620482444763 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33695369958877563 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17829057574272156 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5585705041885376 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4286320209503174 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1359943151473999 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004961125086992979 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15683121979236603 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7200000286102295 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_4_of_4‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9751313328742981 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9743407964706421 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14368762075901031 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009260655380785465 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21941259503364563 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9312669038772583 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12124933302402496 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6917919516563416 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9190823435783386 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9824514985084534 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.997619092464447 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.4049246609210968 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6830280423164368 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.17629782855510712 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48888537287712097 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.907943844795227 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6656010150909424 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.35550656914711 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04791666939854622 _step‚194 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11877259612083435 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030051227658987045 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2150435447692871 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17974956333637238 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5859659314155579 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14681938290596008 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03625812754034996 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1ÂÅZn=† ‚(:² ÞõÒ½¢Ÿ+# proc.memory.rssMB‚ 2823.14453125 network.sent‚ 3.002733e+06 disk.dm-0.out‚5972.763671875 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚3659.0341796875 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚2715.10302734375* proc.memory.percent‚17.231106758117676 cpu‚26.158650497688715 disk.md0.in‚0 disk.dm-0.in‚0 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 131.62109375 disk.nvme1n1.in‚0 disk.md0.out‚ 31.90234375 disk.dm-1.in‚0! disk.dm-1.out‚333.66845703125$ memory_percent‚99.19664859771729 proc.cpu.threads‚17 network.recv‚320698) disk./.usagePercent‚75.94911009728413% disk./.usageGB‚1166.0087127685547% disk.nvme1n1.out‚2715.10302734375 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚3659.0341796875ņ€š‡ ‚(: ÞõÒ§¯ô+ gpu.0.gpu‚55.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.277' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.75899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·p ,0ˆ + _wandbruntime‚971  _runtime‚971t@!ȉ  ' lr-AdamW/pg1‚0.00010131541683776774 ' lr-AdamW/pg2‚0.00010131541683776774 trainer/global_step‚2449 % _timestamp‚1.7844978975510838e+09 _runtime‚ 971.462038529 _step‚195Á§‡ÍŠ Ç ' lr-AdamW/pg2‚0.00010131541683776774 trainer/global_step‚2449 % _timestamp‚1.7844978975510838e+09 _runtime‚ 971.462038529 _step‚195 ' lr-AdamW/pg1‚0.00010131541683776774Ãm‡—0‹ + _wandbruntime‚971  _runtime‚971áò)À'Œ ºO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10794927 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96199474698240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1724483007488 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94970380514406 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _step‚196 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92020265849344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1724571029504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18652843 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17095114 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24849866 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94474991690752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1774526562816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1724571029504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18652841 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43492564 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17094881 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚2449 _runtime‚ 971.463041781 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96199474698240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18652841 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24839723 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96199579759616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96199579759616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816ˆ ôÅ' ¿O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96199474698240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18652841 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24839723 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96199579759616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96199579759616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10794927 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96199474698240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1724483007488 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94970380514406 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _step‚196 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92020265849344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1724571029504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18652843 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17095114 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24849866 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94474991690752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1774526562816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1724571029504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18652841 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43492564 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17094881 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚2449 _runtime‚ 971.463041781ĨMk€0Ž + _wandbruntime‚971  _runtime‚971ûŠÃþ ÅM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1725249146368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10799217 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96237519175168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1725249146368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96237624243712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92056660426752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96237624243712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18660164 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17101813 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1775228072960 _runtime‚ 971.729262457 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18660162 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24859778 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94512358057984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94512375097344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1775049763840 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43519942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6302365 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90281610662912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6302596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1719775570442 trainer/global_step‚2449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10799214 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95008031721556 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94512358057984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90281635566592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.curl÷ýÍrent‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17101579 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24849621 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43509783 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95007928533610 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43519942 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92056863639552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18660162 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1719861719028 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93288170002528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43519942 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92056863639552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18660162 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1719861719028 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93288170002528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96237519175168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1725249146368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96237624243712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92056660426752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96237624243712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18660164 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17101813 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1775228072960 _runtime‚ 971.729262457 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18660162 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24859778 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94512358057984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94512375097344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1775049763840 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43519942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6302365 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90281610662912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6302596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1719775570442 trainer/global_step‚2449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10799214 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95008031721556 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94512358057984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90281635566592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17101579 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24849621 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43509783 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95007928533610 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712Åm…z™0‘ + _wandbruntime‚971  _runtime‚9718Ö'ñ’ û /loss/FBt1/detection_loss‚1.6825514328609648 trainer/global_step‚2449 _runtime‚ 971.729805468 2loss/SRqqt2/assignment_loss‚0.9913493898302841 0loss/SRqqt2/detection_loss‚0.677165441235884 epoch‚3 .loss/FBt2/detection_loss‚5.796680314007811 &loss/total_loss‚19.669442993415174 0loss/FRt2/assignment_loss‚1.7153195708802662 1loss/SRqqt1/assignment_loss‚1.339483028234099 0loss/SRqqt1/detection_loss‚1.721233724943886 $ _timestamp‚1.784497897819181e+09 _step‚198 0loss/FRt1/assignment_loss‚3.2881276945418105 1loss/FBt1/assignment_loss‚0.34968964664689184 1loss/FBt2/assignment_loss‚0.06965754672676888 /loss/FRt1/detection_loss‚0.5848710074055761 .loss/FRt2/detection_loss‚1.4533141961009338z©“ € 0loss/FRt1/assignment_loss‚3.2881276945418105 1loss/FBt1/assignment_loss‚0.34968964664689184 1loss/FBt2/assignment_loss‚0.06965754672676888 /loss/FRt1/detection_loss‚0.5848710074055761 .loss/FRt2/detection_loss‚1.453314196100933 /loss/FBt1/detection_loss‚1.6825514328609648 trainer/global_step‚2449 _runtime‚ 971.729805468 2loss/SRqqt2/assignment_loss‚0.9913493898302841 0loss/SRqqt2/detection_loss‚0.677165441235884 epoch‚3 .loss/FBt2/detection_loss‚5.796680314007811 &loss/total_loss‚19.669442993415174 0loss/FRt2/assignment_loss‚1.7153195708802662 1loss/SRqqt1/assignment_loss‚1.339483028234099 0loss/SRqqt1/detection_loss‚1.721233724943886 $ _timestamp‚1.784497897819181e+09 _step‚198Æ íu!4” ‚(:© íõÒþíÏ+ disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚2754.005859375 disk.dm-0.in‚0 disk.dm-1.out‚ 359.78515625 network.sent‚ 3.073466e+06% disk./.usageGB‚1165.0081939697266 disk.md0.out‚ 32.30859375 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚99.1969108581543& proc.memory.availableMB‚ 131.578125* proc.memory.percent‚17.231106758117676 network.recv‚325159 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0! disk.dm-0.out‚5987.1044921875 disk.dm-1.in‚0 cpu‚26.155208952148815) disk./.usagePercent‚75.88394033347836& disk.nvme0c0n1.out‚3661.4736328125$ disk.nvme0n1.out‚3661.4736328125# disk.nvme1n1.out‚2754.005859375 disk.md0.in‚0# proc.memory.rssMB‚ 2823.14453125 proc.cpu.threads‚17 disk.nvme0n1.in‚0Uc+›• ‚(: íõÒÿ÷ü+ gpu.0.gpu‚56.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚142.733' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚47.577666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]J)Ü9– ‚(:® üõÒÞÿµ+' disk.nvme0c0n1.out‚3666.96826171875 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2794.54736328125% disk.nvme1n1.out‚2794.54736328125 proc.cpu.threads‚17 network.recv‚328944% disk./.usageGB‚1165.0079231262207 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚3666.96826171875 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 32.5546875 network.sent‚ 3.123504e+06 disk.dm-0.in‚0! disk.dm-0.out‚5992.6904296875# memory_percent‚99.1969108581543& proc.memory.availableMB‚ 131.578125* proc.memory.percent‚17.231106758117676 cpu‚26.151403690958862) disk./.usagePercent‚75.88392269182353 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.14453125 disk.dm-1.out‚ 399.69921875Ó¡ˆ— ‚(:… üõÒ·Ýå+ gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚188.766' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚62.922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K¿RŠ0˜ + _wandbruntime‚991  _runtime‚991‹éÆ™ À & lr-AdamW/pg1‚0.0001012882308210787 & lr-AdamW/pg2‚0.0001012882308210787 trainer/global_step‚2499 % _timestamp‚1.7844979172023544e+09 _runtime‚ 991.113307553 _step‚199&µ8?Ëš Å & lr-AdamW/pg1‚0.0001012882308210787 & lr-AdamW/pg2‚0.0001012882308210787 trainer/global_step‚2499 % _timestamp‚1.7844979172023544e+09 _runtime‚ 991.113307553 _step‚199Çä¢Ù10› + _wandbruntime‚991  _runtime‚991|&höÁ'œ »O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1758508771328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19018891 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92030698271744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2499 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25336475 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1809493442048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1758420749312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1753015242880 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44355366 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1758508771328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96343310052352 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96847916565648 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95094884283408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7844979172033598e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25346618 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98101730801664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93840191713792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17432434 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92030673368064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19018893 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6422776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98101835863040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1809313028608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98101730801664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25346618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11009427 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19018891 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11009424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96847813377814 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152R×Íy  ÀO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95094884283408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7844979172033598e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25346618 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98101730801664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93840191713792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚174324£ªPÌM34 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92030673368064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19018893 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6422776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98101835863040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1809313028608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98101730801664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25346618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11009427 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19018891 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11009424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96847813377814 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1758508771328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19018891 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92030698271744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2499 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25336475 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1809493442048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1758420749312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1753015242880 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44355366 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1758508771328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96343310052352 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96847916565648 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816ÈT‚5&0ž + _wandbruntime‚991  _runtime‚991e¢¦ÑÊ&Ÿ ÄM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25346373 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1810195855360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93876379780096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚201 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11013714 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44372585 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17439205 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1753691194436 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96885361397018 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚2499 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17438947 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92066393428992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6425233 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1810011254784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98139880347136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _runtime‚ 991.38318414 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96380676419584 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96885464584964 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95131756351168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1753605045850 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44382744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44372585 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19026214 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25346373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1759186888192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7844979174720047e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19026214 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25356530 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93876589284352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96380676419584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1759098859008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1759098859008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92066368525312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95131773390528 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44382744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98139880347136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98139775278592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1759186888192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19026212 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17438947 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92066393428992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6425233 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1810011254784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98139880347136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _runtime‚ 991.38318414 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96380676419584 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96885464584964 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95131756351168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1753605045850 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44382744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44372585 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19026214 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25346373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1759186888192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7844979174720047e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19026214 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25356530 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93876589284352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96380676419584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1759098859008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1759098859008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92066368525312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95131773390528 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44382744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98139880347136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98139775278592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1759186888192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19026212 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93876379780096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚201 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11013714 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44372585 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17439205 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1753691194436 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751É»o}0¡ + _wandbruntime‚991  _runtime‚991ÑÖÍ‹¢ ý $ _timestamp‚1.784497917473118e+09 &loss/total_loss‚14.621373196195618 0loss/FBt2/assignment_loss‚0.1421265753609733 /loss/FRt2/detection_loss‚1.2886423794449242 0loss/SRqqt1/detection_loss‚1.730704281065606 1loss/SRqqt2/detection_loss‚0.7291592571917447 _step‚202 0loss/FRt2/assignment_loss‚1.6253320626824734 /loss/FBt1/detection_loss‚1.1278298455004774 trainer/global_step‚2499 /loss/FBt2/detection_loss‚1.5846629844762437 epoch‚3 /loss/FRt1/assignment_loss‚3.165193490646592 2loss/SRqqt1/assignment_loss‚1.2711064788288085 2loss/SRqqt2/assignment_loss‚0.9580950269267319 /loss/FRt1/detection_loss‚0.6440990237648869 _runtime‚ 991.383756461 1loss/FBt1/assignment_loss‚0.35442179030615717+·gx£ ‚ 1loss/FBt1/assignment_loss‚0.35442179030615717 $ _timestamp‚1.784497917473118e+09 &loss/total_loss‚14.621373196195618 0loss/FBt2/assignment_loss‚0.1421265753609733 /loss/FRt2/detection_loss‚1.2886423794449242 0loss/SRqqt1/detection_loss‚1.730704281065606 1loss/SRqqt2/detection_loss‚0.7291592571917447 _step‚202 0loss/FRt2/assignment_loss‚1.6253320626824734 /loss/FBt1/detection_loss‚1.1278298455004774 trainer/global_step‚2499 /loss/FBt2/detection_loss‚1.5846629844762437 epoch‚3 /loss/FRt1/assignment_loss‚3.165193490646592 2loss/SRqqt1/assignment_loss‚1.2711064788288085 2loss/SRqqt2/assignment_loss‚0.9580950269267319 /loss/FRt1/detection_loss‚0.6440990237648869 _runtime‚ 991.383756461ÊÕ>øŠ;¤ ‚(:° ‹ŽõÒž½ÿ+ proc.cpu.threads‚17 network.sent‚ 3.139275e+06) disk./.usagePercent‚75.86846611744767! disk.dm-0.out‚5993.3896484375 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚99.19793605804443( proc.memory.availableMB‚ 131.41015625 network.recv‚332522% disk./.usageGB‚1164.7706260681152 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 cpu‚26.147668809882106 disk.md0.out‚ 32.9453125 disk.dm-0.in‚0! disk.dm-1.out‚399.92919921875# disk.nvme0n1.out‚3668.009765625' disk.nvme1c1n1.out‚2795.26806640625* proc.memory.percent‚17.231106758117676 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚3668.009765625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚2795.26806640625# proc.memory.rssMB‚ 2823.14453125y¯=„š¥ ‚(: ‹ŽõÒÖݬ, gpu.0.gpu‚60.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.028' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.00933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°µNÕ2¦ - _wandbruntime‚1010  _runtime‚1010ϧ7.ɧ à ' lr-AdamW/pg1‚0.00010126023739365945 ' lr-AdamW/pg2‚0.00010126023739365945 trainer/global_step‚2549 % _timestamp‚1.7844979367130752e+09 _runtime‚1010.624017303 _step‚203ªY5Ψ È ' lr-AdamW/pg2‚0.00010126023739365945 trainer/global_step‚2549 % _timestamp‚1.7844979367130752e+09 _runtime‚1010.624017303 _step‚203 ' lr-AdamW/pg1‚0.00010126023739365945ËUÃn2© - _wandbruntime‚1010  _runtime‚1010gnÍ} ª ÀO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1792358491136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98211628413952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98211645453312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100003986905088 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95659888894464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25833227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1844254444032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1786758569814 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11223927 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98725349429056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100004091966464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93815431230464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1786844718288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19384943 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98211645453312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100004091966464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93815456134144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1792446513152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6546441 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96938504710768 _step‚204 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19384943 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11223924 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚96938487671408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100003986905088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚1010.628417301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546194 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95659685674496`˜ñ:Ë'« ÅO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45228313 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17770368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17770118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98211628413952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1844432760320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25833227 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100003986905088 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95659888894464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25833227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1844254444032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1786758569814 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11223927 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98725349429056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100004091966464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93815431230464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1786844718288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19384943 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98211645453312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100004091966464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93815456134144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1792446513152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6546441 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96938504710768 _step‚204 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19384943 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11223924 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚96938487671408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100003986905088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚1010.628417301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546194 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95659685674496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1792358491136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98211628413952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98211645453312 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1845131677696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100042031382016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95696282969088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95696075561984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98249011820544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6548862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100042136450560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1793124630016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1787520669844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100042136450560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98249011820544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98762897448372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19392262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17776819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93851151291392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45245546 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98248994781184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1793036600832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844979369768395e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17777079 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45245546 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19392264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚96975376778528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93851126387712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19392264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25843125 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98248994781184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1787434521258 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1793036600832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚96975359739168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45245546 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19392264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚96975376778528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93851126387712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19392264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25843125 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98248994781184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1787434521258 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1793036600832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚96975359739168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1845131677696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S?DeviceStatsMonitor.on_train_basø8tch_end/allocated_bytes.all.freed‚100042031382016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95696282969088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95696075561984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98249011820544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6548862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100042136450560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1793124630016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1787520669844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100042136450560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98249011820544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98762897448372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19392262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17776819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93851151291392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45245546 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98248994781184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1793036600832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844979369768395e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17777079 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816ͽ÷ñû2¯ - _wandbruntime‚1010  _runtime‚1010QÇØ° û % _timestamp‚1.7844979369777863e+09 /loss/FRt2/assignment_loss‚1.632490128958264 2loss/SRqqt2/assignment_loss‚0.8484027566966271 /loss/FRt1/detection_loss‚0.5948701232464826 -loss/FRt2/detection_loss‚1.38515410827513 1loss/SRqqt1/detection_loss‚2.0418186940151397 1loss/SRqqt2/detection_loss‚0.7034090502902608 epoch‚3 /loss/FRt1/assignment_loss‚3.079481787794265 /loss/FBt1/detection_loss‚1.2854040014175117 .loss/FBt2/detection_loss‚2.057605253169086 &loss/total_loss‚15.213632233639915 trainer/global_step‚2549 _runtime‚ 1010.88841205 1loss/SRqqt1/assignment_loss‚1.211196875068586 _step‚206 1loss/FBt1/assignment_loss‚0.24208514830978908 1loss/FBt2/assignment_loss‚0.13171430639877366k¬± € /loss/FRt2/assignment_loss‚1.632490128958264 2loss/SRqqt2/assignment_loss‚0.8484027566966271 /loss/FRt1/detection_loss‚0.5948701232464826 -loss/FRt2/detection_loss‚1.38515410827513 1loss/SRqqt1/detection_loss‚2.0418186940151397 1loss/SRqqt2/detection_loss‚0.7034090502902608 epoch‚3 /loss/FRt1/assignment_loss‚3.079481787794265 /loss/FBt1/detection_loss‚1.2854040014175117 .loss/FBt2/detection_loss‚2.057605253169086 &loss/total_loss‚15.213632233639915 trainer/global_step‚2549 _runtime‚ 1010.88841205 1loss/SRqqt1/assignment_loss‚1.211196875068586 _step‚206 1loss/FBt1/assignment_loss‚0.24208514830978908 1loss/FBt2/assignment_loss‚0.13171430639877366 % _timestamp‚1.7844979369777863e+09Îûi};² ‚(:° šŽõÒØë•, disk.dm-0.in‚0" disk.dm-0.out‚6017.44580078125 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚3671.79443359375% disk.nvme0n1.out‚3671.79443359375 network.sent‚ 3.201757e+06% disk./.usageGB‚1163.7461547851562 disk.nvme1c1n1.in‚0$ memory_percent‚99.19853210449219$ proc.memory.availableMB‚131.3125 disk.nvme1n1.in‚0 disk.md0.out‚ 33.171875 disk.dm-1.in‚0! disk.dm-1.out‚400.10498046875' disk.nvme1c1n1.out‚2816.22705078125# proc.memory.rssMB‚ 2823.14453125 network.recv‚336655) disk./.usagePercent‚75.80173618532201 disk.rbd0.in‚0 disk.nvme0n1.in‚0* proc.memory.percent‚17.231106758117676 cpu‚26.149145288376562 proc.cpu.threads‚17 disk.rbd0.out‚0% disk.nvme1n1.out‚2816.22705078125 disk.md0.in‚0ÛMÛ×™³ ‚(:Ž šŽõÒÕæ¬+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.526' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.50866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚµcY2´ - _wandbruntime‚1029  _runtime‚1029U4ûȵ  ' lr-AdamW/pg2‚0.00010123143700388155 trainer/global_step‚2599 $ _timestamp‚1.784497955471358e+09 _runtime‚1029.382304459 _step‚207 ' lr-AdamW/pg1‚0.00010123143700388155u¿²nͶ Ç ' lr-AdamW/pg1‚0.00010123143700388155 ' lr-AdamW/pg2‚0.00010123143700388155 trainer/global_step‚2599 $ _timestamp‚1.784497955471358e+09 _runtime‚1029.382304459 _step‚207Ï®Âú2· - _wandbruntime‚1029  _runtime‚1029{\ ¾Ì'¸ ÆO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19750993 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100079963814912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98782091059408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26329979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101906243008512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18108105 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100602782292464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826296232960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7844979554722748e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46080970 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19750993 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6669678 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101906348069888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26340122 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚1029.383425401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100602679104630 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101906348069888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100079963814912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826296232960 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11438424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46091115 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97479613092864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18107845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1820674193696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826384254976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97479401484288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19750991 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19750991 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11438427 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6669421 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101906243008512 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100079946775552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330º='WÑ'¹ ËO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100602679104630 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101906348069888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100079963814912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826296232960 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11438424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46091115 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97479613092864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18107845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1820674193696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826384254976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97479401484288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19750991 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19750991 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11438427 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6669421 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101906243008512 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100079946775552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19750993 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100079963814912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98782091059408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26329979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101906243008512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18108105 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100602782292464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826296232960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7844979554722748e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46080970 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19750993 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6669678 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101906348069888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26340122 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚1029.383425401Ð,6¶m2º - _wandbruntime‚1029  _runtime‚1029¥$ù»» ÐM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1821350145252 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6671808 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97515999020544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100640227123834 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19758312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚209 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46108348 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18114522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46098189 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100117313142784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚2599 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101944287485440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26350034 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18114783 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98818980166528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1880089866752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19758314 QBDeviceStatsMonitor.on_traiú\1n_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11442714 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95635884250112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7844979557264268e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100117313142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826974342656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26339877 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46108348 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1879913654784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100117330182144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26339877 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101944392553984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _runtime‚1029.637505595 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101944392553984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97515797904896 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95635909153792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100640330311780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100117330182144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19758312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826974342656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6672066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827062371840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1821263996666 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19758314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26350034 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1880089866752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19758314 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11442714 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95635884250112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7844979557264268e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100117313142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826974342656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26339877 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46108348 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1879913654784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100117330182144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26339877 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101944392553984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _runtime‚1029.637505595 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101944392553984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97515797904896 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95635909153792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100640330311780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100117330182144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19758312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826974342656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6672066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827062371840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1821263996666 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19758314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26350034 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6671808 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97515999020544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100640227123834 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19758312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚209 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46108348 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18114522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46098189 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100117313142784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚2599Ñ×÷Üw2½ - _wandbruntime‚1029  _runtime‚1029D%*¯¾ ü trainer/global_step‚2599 -loss/FBt2/detection_loss‚1.11412992245505 .loss/FRt1/assignment_loss‚3.20375385952806 1loss/FBt1/assignment_loss‚0.14777897263394252 1loss/FBt2/assignment_loss‚0.11675452926759798 1loss/SRqqt1/detection_loss‚1.5110541358193819 % _timestamp‚1.7844979557273176e+09 _runtime‚1029.637997554 &loss/total_loss‚15.122586954399253 epoch‚3 _step‚210 1loss/SRqqt2/assignment_loss‚1.016170608126679 /loss/FRt1/detection_loss‚0.5400171561765109 /loss/FRt2/detection_loss‚1.4722655646562932 1loss/SRqqt2/detection_loss‚0.9671722021308591 /loss/FRt2/assignment_loss‚1.749232670535212 2loss/SRqqt1/assignment_loss‚1.2466338481754065 /loss/FBt1/detection_loss‚2.0376234848942603VV…¿  /loss/FBt1/detection_loss‚2.0376234848942603 trainer/global_step‚2599 -loss/FBt2/detection_loss‚1.11412992245505 .loss/FRt1/assignment_loss‚3.20375385952806 1loss/FBt1/assignment_loss‚0.14777897263394252 1loss/FBt2/assignment_loss‚0.11675452926759798 1loss/SRqqt1/detection_loss‚1.5110541358193819 % _timestamp‚1.7844979557273176e+09 _runtime‚1029.637997554 &loss/total_loss‚15.122586954399253 epoch‚3 _step‚210 1loss/SRqqt2/assignment_loss‚1.016170608126679 /loss/FRt1/detection_loss‚0.5400171561765109 /loss/FRt2/detection_loss‚1.4722655646562932 1loss/SRqqt2/detection_loss‚0.9671722021308591 /loss/FRt2/assignment_loss‚1.749232670535212 2loss/SRqqt1/assignment_loss‚1.2466338481754065Ò°àÅôšÀ ‚(: ©ŽõÒ¥ïË+ gpu.0.gpu‚67.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚194.468' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.82266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=)L96Á ‚(:« ©ŽõÒ¸âÞ+ disk.md0.in‚0 disk.md0.out‚ 33.4921875 disk.dm-1.in‚0% disk.nvme0n1.out‚3672.70556640625 cpu‚26.14607929520075 proc.cpu.threads‚17 disk.dm-1.out‚ 400.390625 network.sent‚ 3.263932e+06 network.recv‚341418 disk.nvme1n1.in‚0# disk.nvme1n1.out‚2816.935546875 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚3672.70556640625 disk.nvme0n1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.14453125* proc.memory.percent‚17.231106758117676" disk.dm-0.out‚6018.08251953125% disk./.usageGB‚1163.7461128234863 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚2816.935546875 disk.dm-0.in‚0 disk.rbd0.in‚0$ memory_percent‚99.20036792755127( proc.memory.availableMB‚ 131.01171875) disk./.usagePercent‚75.80173345210788ㆫï2 - _wandbruntime‚1047  _runtime‚1047o;+ÊÉà à ' lr-AdamW/pg1‚0.00010120183011304171 ' lr-AdamW/pg2‚0.00010120183011304171 trainer/global_step‚2649 % _timestamp‚1.7844979738306882e+09 _runtime‚1047.741584392 _step‚211ƒ«ÒÎÄ È ' lr-AdamW/pg1‚0.00010120183011304171 ' lr-AdamW/pg2‚0.00010120183011304171 trainer/global_step‚2649 % _timestamp‚1.7844979738306882e+09 _runtime‚1047.741584392 _step‚211ÓGÁõ2Å - _wandbruntime‚1047  _runtime‚1047`!*$Î'Æ ÈO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102480111968038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100625711486768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1860233974784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1860321996800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97384971858944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1854503669104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11652924 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1860321996800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18445633 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1854417520630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚2649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103808499111936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1047.742321343 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101948265137152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26826731 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100625694447408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1914187210752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20117041 _step‚212 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101948282176512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46953917 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99299134166016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103808604173312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103808499111936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1914365527040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102480215155872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6792709 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751}yÛ Ç ÍO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102480111968038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100625711486768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1860233974784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1860321996800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_starÈüóøt/requested_bytes.all.current‚ 103187834 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97384971858944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1854503669104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11652924 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1860321996800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18445633 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1854417520630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚2649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103808499111936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1047.742321343 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101948265137152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26826731 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100625694447408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1914187210752 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20117041 _step‚212 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101948282176512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46953917 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99299134166016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103808604173312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103808499111936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1914365527040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102480215155872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6792709 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016Ôš³‹2È - _wandbruntime‚1047  _runtime‚1047éÐØ&É ÒM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100662566515168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26836629 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1860912084480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7844979740849664e+09 _runtime‚1047.996077086 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99335733070336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46971150 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚103846543588864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20124364 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26846786 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚2649 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26846786 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1914881453568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101985631504384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1861000113664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11657217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103846648657408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18452499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1860912084480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18452748 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102517659987242 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20124362 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100662583554528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚213 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46971150 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1855179620660 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1855093472074 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0‡pMŸÝ&Ê ×M E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100662566515168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26836629 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1860912084480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7844979740849664e+09 _runtime‚1047.996077086 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99335733070336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46971150 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚103846543588864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20124364 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26846786 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚2649 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26846786 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1914881453568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101985631504384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1861000113664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11657217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103846648657408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18452499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1860912084480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18452748 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102517659987242 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20124362 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100662583554528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚213 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46971150 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1855179620660 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1855093472074ÕîÄÁ2Ë - _wandbruntime‚1047  _runtime‚10477©šÌ € 0loss/FRt1/assignment_loss‚3.1452346006843532 2loss/SRqqt1/assignment_loss‚1.1301803333684801 1loss/FBt2/assignment_loss‚0.10279724690919667 1loss/SRqqt2/detection_loss‚0.7913914015003451 .loss/FBt1/detection_loss‚1.411067128992516 /loss/FBt2/detection_loss‚1.1669780540693424 _runtime‚1047.997636149 0loss/FRt2/assignment_loss‚1.7299874600242167 1loss/FBt1/assignment_loss‚0.15962110981376226 /loss/FRt2/detection_loss‚1.4118395054232813 trainer/global_step‚2649 _step‚214 epoch‚3 2loss/SRqqt2/assignment_loss‚1.0092729266864209 /loss/FRt1/detection_loss‚0.5624185842498604 1loss/SRqqt1/detection_loss‚1.3702607162650176 &loss/total_loss‚13.991049067986793 $ _timestamp‚1.784497974086581e+09èQÑÊ Í … 0loss/FRt1/assignment_loss‚3.1452346006843532 2loss/SRqqt1/assignment_loss‚1.1301803333684801 1loss/FBt2/assignment_loss‚0.10279724690919667 1loss/SRqqt2/detection_loss‚0.7913914015003451 .loss/FBt1/detection_loss‚1.411067128992516 /loss/FBt2/detection_loss‚1.1669780540693424 _runtime‚1047.997636149 0loss/FRt2/assignment_loss‚1.7299874600242167 1loss/FBt1/assignment_loss‚0.15962110981376226 /loss/FRt2/detection_loss‚1.4118395054232813 trainer/global_step‚2649 _step‚214 epoch‚3 2loss/SRqqt2/assignment_loss‚1.0092729266864209 /loss/FRt1/detection_loss‚0.5624185842498604 1loss/SRqqt1/detection_loss‚1.3702607162650176 &loss/total_loss‚13.991049067986793 $ _timestamp‚1.784497974086581e+09ÖÊ ìÝ+Î ‚(:  ¸ŽõÒßΦ+ disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17 network.recv‚345669& disk.nvme1c1n1.out‚2818.9326171875$ disk.nvme1n1.out‚2818.9326171875 disk.md0.out‚33.6875 disk.dm-0.out‚ 6028.84375 disk.rbd0.out‚0$ disk.nvme0n1.out‚3681.9560546875$ memory_percent‚99.19908046722412 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 131.22265625# proc.memory.rssMB‚ 2823.14453125 disk.dm-1.out‚ 400.453125 network.sent‚ 3.313283e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚0& disk.nvme0c0n1.out‚3681.9560546875* proc.memory.percent‚17.231106758117676 cpu‚26.14498137036977' disk./.usagePercent‚75.801735688374% disk./.usageGB‚1163.7461471557617”è@™Ï ‚(:Ž ¸ŽõÒšèü+ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚189.718' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.23933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Þlú2Ð ‚(:§ ÇŽõÒóĽ+( proc.memory.availableMB‚ 131.19140625 proc.cpu.threads‚17% disk./.usageGB‚1163.7539978027344 disk.md0.out‚ 34.015625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2823.14453125* proc.memory.percent‚17.231106758117676 network.sent‚ 3.376565e+06 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚3683.3896484375 disk.nvme1c1n1.in‚0 cpu‚26.148984102904123( disk./.usagePercent‚75.8022470478902$ disk.nvme1n1.out‚2819.5107421875 disk.md0.in‚0 disk.dm-0.out‚ 6029.99609375 disk.dm-1.out‚ 400.6015625 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚3683.3896484375# memory_percent‚99.1992712020874 network.recv‚350477 disk.nvme1n1.in‚0 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚2819.5107421875éà\,šÑ ‚(: ÇŽõÒ¾ùô+ gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚142.802' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚47.60066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±2†³2Ò - _wandbruntime‚1066  _runtime‚1066¹âG’ÈÓ Â ' lr-AdamW/pg1‚0.00010117141719535436 ' lr-AdamW/pg2‚0.00010117141719535436 trainer/global_step‚2699 $ _timestamp‚1.784497992610759e+09 _runtime‚1066.521689469 _step‚215-“'ÍÔ Ç ' lr-AdamW/pg1‚0.00010117141719535436 ' lr-AdamW/pg2‚0.00010117141719535436 trainer/global_step‚2699 $ _timestamp‚1.784497992610759e+09 _runtime‚1066.521689469 _step‚215×Jóì©2Õ - _wandbruntime‚1066  _runtime‚1066Lá^gåÖ ÉO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103816583498752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27333626 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47806574 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27323483 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocatedŒ]Òmê"_bytes.small_pool.allocated‚ 1894259738624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20483091 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99169729721344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102469297835408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105710860276736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47816719 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1888246996038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6916293 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1888333144512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18783468 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1949301715456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1949121302016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105710755215360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27323483 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18783720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1894259738624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105710755215360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101119031436800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1894171716608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20483093 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11867424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103816600538112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784497992612643e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103816583498752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20483091 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104357544831446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1894171716608 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104357648019280¨˜ïÔ'× ÎO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1894171716608 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104357648019280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103816583498752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27333626 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47806574 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27323483 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1894259738624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20483091 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99169729721344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102469297835408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105710860276736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47816719 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1888246996038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6916293 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1888333144512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18783468 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1949301715456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1949121302016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105710755215360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27323483 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18783720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1894259738624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105710755215360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101119031436800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1894171716608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20483093 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11867424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103816600538112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784497992612643e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103816583498752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20483091 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104357544831446ØÈ ×2Ø - _wandbruntime‚1066  _runtime‚1066ƒÉèHÚ&Ù ÔM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47823793 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚101155212484608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27333381 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11871717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20490414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27333381 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103853966905344 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6918460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20490412 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20490412 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27343538 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103853949865984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18790174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11871714 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99205399974912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1894849826304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47833952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚1066.786196821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TaÚ ÙM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27333381 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11871717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20490414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27333381 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103853966905344 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6918460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20490412 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20490412 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27343538 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103853949865984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18790174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11871714 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99205399974912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1894849826304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47833952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚1066.786196821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47823793 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚101155212484608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482Ù¼p92Û - _wandbruntime‚1066  _runtime‚1066©Â±hÜ ÿ 1loss/FBt2/assignment_loss‚0.43421473262703025 /loss/FBt1/detection_loss‚2.5450292340406597 %loss/total_loss‚16.76637105027499 2loss/SRqqt2/assignment_loss‚0.9168745865436463 /loss/FRt1/detection_loss‚0.5402475096614237 trainer/global_step‚2699 epoch‚3 0loss/FRt2/assignment_loss‚1.5949272474432303 1loss/SRqqt1/assignment_loss‚1.074980140256661 1loss/SRqqt1/detection_loss‚1.3896206811162481 % _timestamp‚1.7844979928761249e+09 _step‚218 /loss/FRt2/detection_loss‚1.3166971106285357 1loss/SRqqt2/detection_loss‚0.8568466992286369 .loss/FBt2/detection_loss‚2.716770495271453 _runtime‚1066.790375668 0loss/FRt1/assignment_loss‚3.2426951951358145 1loss/FBt1/assignment_loss‚0.13746741832164844ç‘Õ Ý „ epoch‚3 0loss/FRt2/assignment_loss‚1.5949272474432303 1loss/SRqqt1/assignment_loss‚1.074980140256661 1loss/SRqqt1/detection_loss‚1.3896206811162481 % _timestamp‚1.7844979928761249e+09 _step‚218 /loss/FRt2/detection_loss‚1.3166971106285357 1loss/SRqqt2/detection_loss‚0.8568466992286369 .loss/FBt2/detection_loss‚2.716770495271453 _runtime‚1066.790375668 0loss/FRt1/assignment_loss‚3.2426951951358145 1loss/FBt1/assignment_loss‚0.13746741832164844 1loss/FBt2/assignment_loss‚0.43421473262703025 /loss/FBt1/detection_loss‚2.5450292340406597 %loss/total_loss‚16.76637105027499 2loss/SRqqt2/assignment_loss‚0.9168745865436463 /loss/FRt1/detection_loss‚0.5402475096614237 trainer/global_step‚2699ÚE›Íè*Þ ‚(:Ÿ ÖŽõÒÄÓ‚," disk.nvme0n1.out‚ 3689.01171875" disk.nvme1n1.out‚ 2848.51171875# proc.memory.rssMB‚ 2823.14453125 proc.cpu.threads‚17% disk./.usageGB‚1163.8737983703613 disk.md0.out‚ 34.37109375$ disk.nvme0c0n1.out‚ 3689.01171875( proc.memory.availableMB‚ 131.12890625 cpu‚26.164523081718503 disk.nvme0c0n1.in‚0 network.sent‚ 3.394277e+06 network.recv‚353887 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 400.734375 disk.nvme1n1.in‚0$ memory_percent‚99.19965267181396 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 2848.51171875* proc.memory.percent‚17.231106758117676) disk./.usagePercent‚75.81005037422969 disk.dm-0.out‚ 6063.71875 disk.rbd0.out‚0 disk.nvme0n1.in‚0¥ñ2¦œß ‚(:‘ ÖŽõÒ¶˜±, gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚185.831' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚61.943666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.)ºI2à - _wandbruntime‚1086  _runtime‚1086(Ö‚ÐÇá Á ' lr-AdamW/pg2‚0.00010114019873794415 trainer/global_step‚2749 $ _timestamp‚1.784498012496854e+09 _runtime‚ 1086.40848744 _step‚219 ' lr-AdamW/pg1‚0.00010114019873794415È"ÌÌâ Æ ' lr-AdamW/pg1‚0.00010114019873794415 ' lr-AdamW/pg2‚0.00010114019873794415 trainer/global_step‚2749 $ _timestamp‚1.784498012496854e+09 _runtime‚ 1086.40848744 _step‚219ÛZ^#2ã - _wandbruntime‚1086  _runtime‚1086<“ œÑ'ä ËO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1928109458432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19121187 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104312901223408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107613116380160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104312918262768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102938722320896 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107613011318784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12081927 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚2749 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1928109458432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106234977694854 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928197480448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100954487583744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7844980124976082e+09 _step‚220 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20849143 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7039263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102938519100928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106235080882688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107613011318784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 1086.40854276 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27830378 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27820235 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1984056420864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1928197480448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1922076471446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27820235 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20849141 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19121437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1922162619920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20849141 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105684901860352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105684918899712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816y °°Ö'å ÐO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1928109458432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19121187 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104312901223408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107613116380160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104312918262768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102938722320896 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107613011318784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12081927 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚2749 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1928109458432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106234977694854 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928197480448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100954487583744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7844980124976082e+09 _step‚220 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20849143 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7039263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102938519100928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106235080882688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107613011318784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 1086.40854276 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27830378 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27820235 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1984056420864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1928197480448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1922076471446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27820235 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20849141 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19121437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1922162619920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20849141 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105684901860352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105684918899712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816ÜÆLjÜ2æ - _wandbruntime‚1086  _runtime‚1086Ú§²û ç ÖM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107651160864256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100990182740992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27830133 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1928875597312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7041653 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107651055795712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1928875597312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48696754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.fr¨½DFáeed‚19127867 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1984941654016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106272525714058 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48696754 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928787568128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105722285266944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27840290 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27830133 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20856464 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20856462 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20856462 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19128114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1984750761984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102974908599296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928787568128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106272628902004 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1922838571476 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48686595 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100990157837312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100990157837312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27830133 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1928875597312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7041653 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107651055795712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1928875597312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48696754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19127867 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1984941654016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106272525714058 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48696754 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928787568128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105722285266944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27840290 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27830133 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20856464 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20856462 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20856462 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19128114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1984750761984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102974908599296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928787568128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106272628902004 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482Ý#kg2é - _wandbruntime‚1086  _runtime‚1086Ó/ßê ÿ /loss/FRt1/detection_loss‚0.4750829241820611 0loss/FRt2/assignment_loss‚1.5938878063295707 /loss/FBt1/detection_loss‚1.4949441164189163 trainer/global_step‚2749 _runtime‚1086.676796334 2loss/SRqqt1/assignment_loss‚1.0279359190502542 1loss/SRqqt1/detection_loss‚1.8816213592971591 &loss/total_loss‚15.394359077755794 % _timestamp‚1.7844980127661633e+09 _step‚222 epoch‚3 2loss/SRqqt2/assignment_loss‚0.8626734778688381 1loss/FBt1/assignment_loss‚0.20803814793436004 0loss/FBt2/assignment_loss‚0.2280565380723811 /loss/FRt2/detection_loss‚1.3178630016431152 1loss/SRqqt2/detection_loss‚0.9038200453919966 .loss/FBt2/detection_loss‚2.213897284687711 /loss/FRt1/assignment_loss‚3.186538456879432ò ÷ž ë „ 0loss/FRt2/assignment_loss‚1.5938878063295707 /loss/FBt1/detection_loss‚1.4949441164189163 trainer/global_step‚2749 _runtime‚1086.676796334 2loss/SRqqt1/assignment_loss‚1.0279359190502542 1loss/SRqqt1/detection_loss‚1.8816213592971591 &loss/total_loss‚15.394359077755794 % _timestamp‚1.7844980127661633e+09 _step‚222 epoch‚3 2loss/SRqqt2/assignment_loss‚0.8626734778688381 1loss/FBt1/assignment_loss‚0.20803814793436004 0loss/FBt2/assignment_loss‚0.2280565380723811 /loss/FRt2/detection_loss‚1.3178630016431152 1loss/SRqqt2/detection_loss‚0.9038200453919966 .loss/FBt2/detection_loss‚2.213897284687711 /loss/FRt1/assignment_loss‚3.186538456879432 /loss/FRt1/detection_loss‚0.4750829241820611ÞÂT Ö6ì ‚(:« åŽõÒóÞí+ network.recv‚358606 disk.md0.in‚0 disk.rbd0.in‚0% disk.nvme1n1.out‚2876.88525390625# proc.memory.rssMB‚ 2823.14453125 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚3689.9833984375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2876.88525390625* proc.memory.percent‚17.231106758117676 proc.cpu.threads‚17% disk./.usageGB‚1164.0559577941895 disk.md0.out‚34.84375 disk.dm-0.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚3689.9833984375 disk.nvme1n1.in‚0$ memory_percent‚99.19993877410889( proc.memory.availableMB‚ 131.08203125 network.sent‚ 3.455453e+06( disk./.usagePercent‚75.8219155052395 disk.dm-0.out‚ 6091.82421875! disk.dm-1.out‚400.96044921875 disk.nvme0c0n1.in‚0 cpu‚26.1717797731056³oø›í ‚(: åŽõÒÜÿ•, gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.421' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.80699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q'®†2î - _wandbruntime‚1106  _runtime‚1106î=¢×Åï ¿ _step‚223 % lr-AdamW/pg1‚0.000101108175240838 % lr-AdamW/pg2‚0.000101108175240838 trainer/global_step‚2799 % _timestamp‚1.7844980322285771e+09 _runtime‚1106.1403845925±´*Êð Ä _runtime‚1106.140384592 _step‚223 % lr-AdamW/pg1‚0.000101108175240838 % lr-AdamW/pg2‚0.000101108175240838 trainer/global_step‚2799 % _timestamp‚1.7844980322285771e+09ßÉWœ¥2ñ - _wandbruntime‚1106  _runtime‚1106sD¨Ò'ò ÌO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7162193 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1955992095328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 % _timestamp‚1.7844980322293801e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107553220221952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21215193 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104758442801664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108112410558262 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106156504611408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21215193 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12296424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28316987 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2019197355520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28327130 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1955905946854 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚2799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28327130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107553220221952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚1106.140445972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12296427 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109515372483584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102739245446144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21215191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1962135222272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49542323 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7162444 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1962135222272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104758233290240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28316987 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19458617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109515267422208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108112513746096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Hõ¥ ó ÑO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7162444 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1962135222272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104758233290240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28316987 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19458617 QIDeviceStatsMonito_øÆAË r.on_train_batch_start/inactive_split.small_pool.current‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109515267422208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108112513746096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7162193 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1955992095328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 % _timestamp‚1.7844980322293801e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107553220221952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21215193 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104758442801664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108112410558262 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106156504611408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21215193 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12296424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28316987 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2019197355520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28327130 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1955905946854 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚2799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28327130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107553220221952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚1106.140445972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12296427 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109515372483584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102739245446144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21215191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1962135222272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49542323 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482àUÎ7z2ô - _wandbruntime‚1106  _runtime‚1106¶N †Ü&õ ÖM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28337042 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7844980324977024e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19465417 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109553416967680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109553311899136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49559556 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28326885 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102774940603392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49549397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104794838494208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107590603628544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2019897890816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1962813339136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107590586589184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 trainer/global_step‚2799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1956668046884 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28337042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107590586589184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1962725309952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1956581898298 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49559556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21222514 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12300714 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21222512¡§Rá&ö ÛM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12300717 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107590603628544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2019897890816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1962813339136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107590586589184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 trainer/global_step‚2799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1956668046884 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28337042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107590586589184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1962725309952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1956581898298 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49559556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21222514 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12300714 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21222512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28337042 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7844980324977024e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19465417 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109553416967680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109553311899136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49559556 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28326885 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102774940603392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49549397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104794838494208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@Device]÷^„ö#StatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108000107999408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50405125 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7844980508487957e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1996072964096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19796915 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109421538583552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1989735422262 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21581241 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111417523525632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111417628587008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1995984942080 _step‚228 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0WHfÁÖ' ÐO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109989843421670 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104524003308544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28813739 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1989821570736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106577962040320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28823882 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2053983635456 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12510924 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111417628587008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28823882 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1996072964096 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28813739 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1995984942080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21581241 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7285988 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7285735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111417628587008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1995984942080 _step‚228 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108000125038768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50394980 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109421555622912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108000107999408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50405125 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7844980508487957e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1996072964096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19796915 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109421538583552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1989735422262 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21581241 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111417523525632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50422358 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21588564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1990497522292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1996663051776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2054869597184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚229 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1996751080960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50412199 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7288049 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19803263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110027494628820 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28823637 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28823637 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21588564 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109458921990144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1996751080960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28833794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106614568062976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110027391440874 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104559673562112 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104559698465792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1125.013397793 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2054680802304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108036980067168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374i (%† ÚM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110027494628820 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28823637 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28823637 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21588564 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109458921990144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1996751080960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28833794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106614568062976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110027391440874 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104559673562112 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMoni/Í‘Ÿ»tor.on_train_batch_end/active_bytes.small_pool.current‚88029184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104559698465792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1125.013397793 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2054680802304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108036980067168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108036997106528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50422358 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21588564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1990497522292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1996663051776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2054869597184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚229 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1996751080960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50412199 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7288049 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19803263 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248åƒ32‡ - _wandbruntime‚1125  _runtime‚1125ÐÕ2µˆ þ trainer/global_step‚2849 _runtime‚1125.013948314 /loss/FRt1/assignment_loss‚2.967935234171535 0loss/FRt2/assignment_loss‚1.4883644394694067 2loss/SRqqt2/assignment_loss‚0.8365780326331943 .loss/FRt1/detection_loss‚0.558113484495344 1loss/SRqqt1/detection_loss‚1.6634413348710295 .loss/FBt2/detection_loss‚7.134116995125562 epoch‚3 /loss/FRt2/detection_loss‚1.2847775213092811 1loss/SRqqt2/detection_loss‚0.7876584840390262 .loss/FBt1/detection_loss‚5.722578691929011 _step‚230 1loss/FBt1/assignment_loss‚0.11304136847665038 2loss/FBt2/assignment_loss‚0.020607969731386358 %loss/total_loss‚23.30397721926716 % _timestamp‚1.7844980511033454e+09 2loss/SRqqt1/assignment_loss‚0.7267636630157337í…ŸZ ‰ ƒ /loss/FRt2/detection_loss‚1.2847775213092811 1loss/SRqqt2/detection_loss‚0.7876584840390262 .loss/FBt1/detection_loss‚5.722578691929011 _step‚230 1loss/FBt1/assignment_loss‚0.11304136847665038 2loss/FBt2/assignment_loss‚0.020607969731386358 %loss/total_loss‚23.30397721926716 % _timestamp‚1.7844980511033454e+09 2loss/SRqqt1/assignment_loss‚0.7267636630157337 trainer/global_step‚2849 _runtime‚1125.013948314 /loss/FRt1/assignment_loss‚2.967935234171535 0loss/FRt2/assignment_loss‚1.4883644394694067 2loss/SRqqt2/assignment_loss‚0.8365780326331943 .loss/FRt1/detection_loss‚0.558113484495344 1loss/SRqqt1/detection_loss‚1.6634413348710295 .loss/FBt2/detection_loss‚7.134116995125562 epoch‚3æÿÄ#<Š ‚(:± ’õÒÝȧ+ network.recv‚371249% disk./.usageGB‚1166.2757759094238' disk.nvme1c1n1.out‚3904.89990234375% disk.nvme1n1.out‚3904.89990234375 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.availableMB‚66.3125# proc.memory.rssMB‚ 2823.14453125 network.sent‚ 3.619992e+06 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0! disk.dm-1.out‚405.04052734375) disk./.usagePercent‚75.96650551437381% disk.nvme0n1.out‚5213.74951171875 disk.md0.out‚ 35.94140625" disk.dm-0.out‚8637.14794921875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚5213.74951171875 disk.md0.in‚0$ memory_percent‚99.59526062011719* proc.memory.percent‚17.231106758117676 cpu‚26.167486955228274 proc.cpu.threads‚17‹ˆÁ‹ ‚(:„ ’õÒ Ì„, gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚116.523' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.841 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I³É¶2Œ - _wandbruntime‚1143  _runtime‚11438£ šÆ À & lr-AdamW/pg1‚0.0001010417151921093 & lr-AdamW/pg2‚0.0001010417151921093 trainer/global_step‚2899 % _timestamp‚1.7844980697326047e+09 _runtime‚ 1143.64536878 _step‚231°z˜ËŽ Å % _timestamp‚1.7844980697326047e+09 _runtime‚ 1143.64536878 _step‚231 & lr-AdamW/pg1‚0.0001010417151921093 & lr-AdamW/pg2‚0.0001010417151921093 trainer/global_step‚2899ç=ÄPü2 - _wandbruntime‚1143  _runtime‚1143%ËŽ+Ñ' ËO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚232 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29310491 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51267927 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7409237 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108397719230464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106308761170944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2030010705920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111867379472912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2023564897670 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108397924547584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111289873984512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2030010705920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111289856945152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20134661 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111867276285078 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51257782 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113319779629056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113319884690432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2029922683904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21947291 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21947291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113319779629056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2088982963200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20134913 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12725424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12725427 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111289856945152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7409486 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106308736267264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113319884690432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2089163376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚ 1143.64546077 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 % _timestamp‚1.7844980697331946e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 % _timestamp‚1.7844980697331946e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚232 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29310491 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51267927 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7409237 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108397719230464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106308761170944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2030010705920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111867379472912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2023564897670 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108397924547584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111289873984512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2030010705920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111289856945152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20134661 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111867276285078 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51257782 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113319779629056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113319884690432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2029922683904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21947291 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21947291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113319779629056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2088982963200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360èÂì£2’ - _wandbruntime‚1143  _runtime‚1143»M1£ “ ÓM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytePà Î"s.small_pool.allocated‚ 2030688822784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108434112910848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2024326997700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111327240351744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2089863989760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21954612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108434320317952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51275001 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20141638 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20141389 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2030600793600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29330546 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29320389 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784498069990964e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2030600793600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113357824105984 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51285160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2089681486336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111904927492228 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29330546 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111904824304282 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2024240849114 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21954612 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113357929174528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51275001 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21954614 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7411675 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29320389 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51285160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111327223312384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7411921 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106344456328192 _runtime‚ 1143.9019997 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111327223312384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109880583455168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330°×Þ&” ØM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2030688822784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12729714 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51285160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2089681486336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111904927492228 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29330546 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111904824304282 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2024240849114 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21954612 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113357929174528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51275001 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21954614 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7411675 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29320389 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51285160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111327223312384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7411921 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106344456328192 _runtime‚ 1143.9019997 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111327223312384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109880583455168 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2030688822784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108434112910848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2024326997700 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111327240351744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2089863989760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21954612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108434320317952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51275001 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20141638 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20141389 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2030600793600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29330546 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29320389 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784498069990964e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2030600793600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113357824105984 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115222035732480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52120584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29807243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123960258048 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7533351 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20473016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110217454387712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚1162.223633233 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113158175306752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115222035732480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22313341 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29817386 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2124144865792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2057480521552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113744709148486 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22313341 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115222140793856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2063860425728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12939927 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108093519033344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113158192346112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111687331814768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113158192346112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12939924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113158175306752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110217663899136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52130729 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29807243 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7533092 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _step‚236 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29817386 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360¶(<-Ÿ ÑO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22313343 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/«ÓˆGª&segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2057394373078 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2063860425728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113744812336320 % _timestamp‚1.7844980883117673e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115222035732480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52120584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29807243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123960258048 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7533351 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20473016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110217454387712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚1162.223633233 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113158175306752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115222035732480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22313341 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29817386 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2124144865792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2057480521552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113744709148486 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22313341 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115222140793856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2063860425728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12939927 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108093519033344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113158192346112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111687331814768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113158192346112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12939924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113158175306752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110217663899136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52130729 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29807243 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7533092 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _step‚236 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29817386 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2063948447744 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 $ _timestamp‚1.784498088567311e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29817141 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2064626564608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113195541673984 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29827298 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20479687 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113782257167690 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52147962 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22320662 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29827298 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115260185277952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52147962 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚237 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12944214 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22320664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52137803 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111724203882528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2064626564608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22320664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110254051960832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111724186843168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115260185277952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2064538535424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7535725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124663655424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20479942 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115260080209408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113195541673984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113782360355636 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚1162.478363608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2064538535424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12944217 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108129189286912m¸à&¢ ÚM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12944214 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22320664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52137803 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111724203882528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2064626564608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22320664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110254051960832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111724186843168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115260185277952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2064538535424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7535725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124663655424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20479942 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115260080209408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113195541673984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113782360355636 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚1162.478363608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2064538535424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12944217 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108129189286912 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 $ _timestamp‚1.784498088567311e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29817141 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2064626564608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113195541673984 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29827298 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20479687 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113782257167690 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52147962 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22320662 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29827298 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115260185277952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52147962 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚237í8ÉØ#2£ - _wandbruntime‚1162  _runtime‚1162<Ö̤ ý 0loss/SRqqt2/assignment_loss‚0.81361996471022 1loss/FBt2/assignment_loss‚0.27361065370856474 /loss/FBt1/detection_loss‚1.6653815879522718 %loss/total_loss‚14.49300457872153 _step‚238 1loss/SRqqt2/detection_loss‚0.7647450373067217 epoch‚3 $ _timestamp‚1.784498088568236e+09 0loss/FRt2/assignment_loss‚1.5450295711320543 /loss/FRt1/detection_loss‚0.7328017230484283 1loss/SRqqt1/detection_loss‚1.7430513555917777 /loss/FBt2/detection_loss‚1.2530000291270746 _runtime‚1162.478825139 /loss/FRt1/assignment_loss‚3.049752312705404 2loss/SRqqt1/assignment_loss‚1.0418108104601165 1loss/FBt1/assignment_loss‚0.22548218585589994 /loss/FRt2/detection_loss‚1.3847193471229975 trainer/global_step‚2949JTŒ¥ ‚ 1loss/SRqqt2/detection_loss‚0.7647450373067217 epoch‚3 $ _timestamp‚1.784498088568236e+09 0loss/FRt2/assignment_loss‚1.5450295711320543 /loss/FRt1/detection_loss‚0.7328017230484283 1loss/SRqqt1/detection_loss‚1.7430513555917777 /loss/FBt2/detection_loss‚1.2530000291270746 _runtime‚1162.478825139 /loss/FRt1/assignment_loss‚3.049752312705404 2loss/SRqqt1/assignment_loss‚1.0418108104601165 1loss/FBt1/assignment_loss‚0.22548218585589994 /loss/FRt2/detection_loss‚1.3847193471229975 trainer/global_step‚2949 0loss/SRqqt2/assignment_loss‚0.81361996471022 1loss/FBt2/assignment_loss‚0.27361065370856474 /loss/FBt1/detection_loss‚1.6653815879522718 %loss/total_loss‚14.49300457872153 _step‚238î¥ØÇÃ/¦ ‚(:¤ °õÒà¾, disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 5225.32421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0* proc.memory.percent‚17.231106758117676 cpu‚26.16215089603155 network.sent‚ 3.745026e+06 disk.dm-0.in‚0$ disk.nvme0c0n1.out‚ 5225.32421875# disk.nvme1n1.out‚4440.994140625 proc.cpu.threads‚17# disk./.usageGB‚1163.55717086792 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.md0.out‚ 36.5546875# memory_percent‚99.5957612991333' proc.memory.availableMB‚ 66.23046875 network.recv‚379089) disk./.usagePercent‚75.78942653430516! disk.dm-0.out‚8648.4462890625 disk.dm-1.in‚0 disk.dm-1.out‚940.0771484375% disk.nvme1c1n1.out‚4440.994140625 disk.md0.in‚0# proc.memory.rssMB‚ 2823.14453125 disk.rbd0.out‚0n‰+š§ ‚(: °õÒ¤Û©, gpu.0.gpu‚55.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.678' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.55933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Cò¨ - _wandbruntime‚11¼¢–80  _runtime‚1180’ÈîÇ© Á _runtime‚1180.804188881 _step‚239 & lr-AdamW/pg1‚0.0001009720413071215 & lr-AdamW/pg2‚0.0001009720413071215 trainer/global_step‚2999 % _timestamp‚1.7844981068932667e+09‚~âç̪ Æ _runtime‚1180.804188881 _step‚239 & lr-AdamW/pg1‚0.0001009720413071215 & lr-AdamW/pg2‚0.0001009720413071215 trainer/global_step‚2999 % _timestamp‚1.7844981068932667e+09︒³ë2« - _wandbruntime‚1180  _runtime‚1180úf?Ò'¬ ÌO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13154424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112037428828160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112037208830976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2097798167552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚2999 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22679391 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7656930 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2091309996960 % _timestamp‚1.7844981068941712e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20811354 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109878276895744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22679391 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30303995 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2097886189568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52983386 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22679393 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30303995 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2158956838912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115622245199728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7657193 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109878251992064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30314138 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2091223848486 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115026510707712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113530935202768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115026493668352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22679393 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117124291835904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115622142011894 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115026493668352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52993531 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117124396897280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2097798167552k´Oj×'­ ÑO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30314138 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2091223848486 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115026510707712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113530935202768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115026493668352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22679393 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117124291835904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115622142011894 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115026493668352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52993531 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117124396897280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2097798167552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13154424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112037428828160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112037208830976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2097798167552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚2999 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22679391 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7656930 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2091309996960 % _timestamp‚1.7844981068941712e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20811354 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109878276895744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22679391 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30303995 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2097886189568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52983386 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22679393 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30303995 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2158956838912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115622245199728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7657193 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109878251992064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330ðôx&2® - _wandbruntime‚1181  _runtime‚1181!×U Ü&¯ ÖM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22686714 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7659379 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2091985948516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2098476277248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115063877074944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30313893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113567790231168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115063877074944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30313893 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109913972052992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2159658270208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113567807270528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115063860035584 trainer/global_step‚2999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2098564306432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20818093 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117162441381376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 % _timestamp‚1.7844981071495507e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2159840773632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53000605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30324050 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22686714 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚1181.060636944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115659793219044 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53010764 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30324050 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53010764 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13158714 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112073812826624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112073605419520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109913947149312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22686712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13158717 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22686712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115063860035584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2098476277248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117162336312832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2091899799930 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20818340 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53000605 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2098564306432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360+(0° ÛM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2098564306432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20818093 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated¯‚ƒ±d‚117162441381376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 % _timestamp‚1.7844981071495507e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2159840773632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53000605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30324050 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22686714 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚1181.060636944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115659793219044 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53010764 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30324050 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53010764 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13158714 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112073812826624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112073605419520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109913947149312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22686712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13158717 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22686712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115063860035584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2098476277248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117162336312832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2091899799930 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20818340 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53000605 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2098564306432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22686714 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7659379 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2091985948516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2098476277248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115063877074944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30313893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113567790231168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115063877074944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30313893 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109913972052992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2159658270208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113567807270528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115063860035584 trainer/global_step‚2999ñlpJ2± - _wandbruntime‚1181  _runtime‚1181¸H’p²  2loss/SRqqt1/assignment_loss‚0.8626730607230534 2loss/SRqqt2/assignment_loss‚0.8806283677844222 /loss/FRt1/detection_loss‚0.6185845554155324 _runtime‚1181.061138275 /loss/FRt1/assignment_loss‚3.044038422500031 1loss/FBt1/assignment_loss‚0.13908401366565215 /loss/FRt2/detection_loss‚1.4496253156969259 1loss/SRqqt1/detection_loss‚1.4590836799242461 1loss/SRqqt2/detection_loss‚0.8702129653981903 &loss/total_loss‚13.697663066175794 epoch‚3 2loss/FBt2/assignment_loss‚0.037104158711432925 _step‚242 0loss/FRt2/assignment_loss‚1.3500158803273237 .loss/FBt1/detection_loss‚2.111848945878064 /loss/FBt2/detection_loss‚0.8747637001509196 trainer/global_step‚2999 % _timestamp‚1.7844981071504633e+09õ(%™ ³ † _step‚242 0loss/FRt2/assignment_loss‚1.3500158803273237 .loss/FBt1/detection_loss‚2.111848945878064 /loss/FBt2/detection_loss‚0.8747637001509196 trainer/global_step‚2999 % _timestamp‚1.7844981071504633e+09 2loss/SRqqt1/assignment_loss‚0.8626730607230534 2loss/SRqqt2/assignment_loss‚0.8806283677844222 /loss/FRt1/detection_loss‚0.6185845554155324 _runtime‚1181.061138275 /loss/FRt1/assignment_loss‚3.044038422500031 1loss/FBt1/assignment_loss‚0.13908401366565215 /loss/FRt2/detection_loss‚1.4496253156969259 1loss/SRqqt1/detection_loss‚1.4590836799242461 1loss/SRqqt2/detection_loss‚0.8702129653981903 &loss/total_loss‚13.697663066175794 epoch‚3 2loss/FBt2/assignment_loss‚0.037104158711432925ò¢ð#9´ ‚(:® ¿õÒß©ú+ disk.rbd0.out‚0& proc.memory.availableMB‚ 66.1640625* proc.memory.percent‚17.231106758117676 proc.cpu.threads‚17 network.recv‚383518 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚5226.3408203125' disk.nvme1c1n1.out‚4441.85400390625 disk.nvme1n1.in‚0 disk.md0.out‚ 37.0859375! disk.dm-1.out‚940.30712890625 network.sent‚ 3.80723e+06) disk./.usagePercent‚75.78942802514923$ disk.nvme0n1.out‚5226.3408203125 disk.dm-0.in‚0! disk.dm-0.out‚8648.9658203125$ memory_percent‚99.59616661071777# proc.memory.rssMB‚ 2823.14453125 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0 cpu‚26.15970599713476% disk./.usageGB‚1163.5571937561035 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚4441.85400390625 disk.dm-1.in‚0iò&b›µ ‚(: ¿õÒÅ…¦, gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚114.973' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.324333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È]á 2¶ - _wandbruntime‚1199  _runtime‚1199½±’³É· à trainer/global_step‚3049 % _timestamp‚1.7844981251452138e+09 _runtime‚1199.056151228 _step‚243 ' lr-AdamW/pg1‚0.00010093600056295033 ' lr-AdamW/pg2‚0.00010093600056295033ÂÏó'θ È ' lr-AdamW/pg1‚0.00010093600056295033 ' lr-AdamW/pg2‚0.00010093600056295033 trainer/global_step‚3049 % _timestamp‚1.7844981251452138e+09 _runtime‚1199.056151228 _step‚243ó-ñl±2¹ - _wandbruntime‚1199  _runtime‚1199W¯Þ²Ò'º ÌO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚1199.056881839 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119026653000704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23045443 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116894829069312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2131823931392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21149935 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21149684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13368927 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117499574875302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23045441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2194158361088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23045441 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116894812029952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53856333 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚244 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2131823931392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30800747 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7780760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119026653000704 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119026547939328 trainer/global_step‚3049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30800747 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115374521551408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7844981251460962e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712€U»Í×'» ÑO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7780760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119026653000704 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119026547939328 trainer/global_step‚3049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30800747 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115374521551408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7844981251460962e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚1199.056881839 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119026653000704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23045443 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116894829069312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2131823931392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21149935 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21149684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13368927 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117499574875302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23045441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2194158361088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23045441 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116894812029952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53856333 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚244 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2131823931392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30800747 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448ô±hÇn2¼ - _wandbruntime‚1199  _runtime‚1199àäöšþ½ ÖM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119064697484800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53863407 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30810645 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21156448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53873566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1199.310933558 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2132502048256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13373214 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7844981253998592e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115411393619168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7783234 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30810645 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2132414019072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7783491 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30820802 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117537122894506 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119064592416256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2132502048256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2125729275338 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116932195436544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13373217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113893589731328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113893373935616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111698705011712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30820802 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚3049 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21156448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53873566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1199.310933558 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2132502048256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13373214 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7844981253998592e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115411393619168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7783234 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30810645 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2132414019072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7783491 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30820802 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117537122894506 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119064592416256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2132502048256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2125729275338 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116932195436544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13373217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113893589731328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113893373935616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111698705011712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30820802 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚3049 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0õÅ^$2¿ - _wandbruntime‚1199  _runtime‚1199sãCzÀ € 0loss/FRt1/assignment_loss‚3.0517184973654903 .loss/FRt1/detection_loss‚0.596962278460308 2loss/SRqqt1/assignment_loss‚0.9101454610454625 2loss/SRqqt2/assignment_loss‚0.7245833269430376 1loss/FBt1/assignment_loss‚0.16959462223732855 /loss/FBt1/detection_loss‚1.0519519302725509 _step‚246 1loss/SRqqt1/detection_loss‚1.3411800617079155 .loss/FBt2/detection_loss‚4.282748997502965 &loss/total_loss‚15.743298326843643 epoch‚3 _runtime‚1199.311455909 0loss/FRt2/assignment_loss‚1.4193901971235114 1loss/FBt2/assignment_loss‚0.06838630985921287 /loss/FRt2/detection_loss‚1.3101510633442204 1loss/SRqqt2/detection_loss‚0.8164855809816396 trainer/global_step‚3049 % _timestamp‚1.7844981254007957e+09×s’— Á … .loss/FBt2/detection_loss‚4.282748997502965 &loss/total_loss‚15.743298326843643 epoch‚3 _runtime‚1199.311455909 0loss/FRt2/assignment_loss‚1.4193901971235114 1loss/FBt2/assignment_loss‚0.06838630985921287 /loss/FRt2/detection_loss‚1.3101510633442204 1loss/SRqqt2/detection_loss‚0.8164855809816396 trainer/global_step‚3049 % _timestamp‚1.7844981254007957e+09 0loss/FRt1/assignment_loss‚3.0517184973654903 .loss/FRt1/detection_loss‚0.596962278460308 2loss/SRqqt1/assignment_loss‚0.9101454610454625 2loss/SRqqt2/assignment_loss‚0.7245833269430376 1loss/FBt1/assignment_loss‚0.16959462223732855 /loss/FBt1/detection_loss‚1.0519519302725509 _step‚246 1loss/SRqqt1/detection_loss‚1.3411800617079155ög@ec? ‚(:´ ÎõÒ¼£+ disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚5233.07958984375$ memory_percent‚99.59635734558105& proc.memory.availableMB‚ 66.1328125 network.recv‚387986' disk.nvme1c1n1.out‚4444.34130859375# proc.memory.rssMB‚ 2823.14453125* proc.memory.percent‚17.231106758117676 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚5233.07958984375 cpu‚26.157556161817325 proc.cpu.threads‚17% disk./.usageGB‚1163.5571746826172 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚4444.34130859375 disk.md0.in‚0 disk.md0.out‚ 37.96484375" disk.dm-0.out‚8656.23291015625! disk.dm-1.out‚940.42431640625 disk.rbd0.out‚0 network.sent‚ 3.856643e+06) disk./.usagePercent‚75.78942678277917Y¨®È›à ‚(: ÎõÒËÛú+ gpu.0.gpu‚51.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚117.344' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.114666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02û9Ä ‚(:® ÝõÒ¤éã+$ memory_percent‚99.59642887115479' proc.memory.availableMB‚ 66.12109375# proc.memory.rssMB‚ 2823.14453125) disk./.usagePercent‚75.78944815154418$ disk./.usageGB‚1163.557502746582 disk.nvme1n1.in‚0 cpu‚26.154998290216565 proc.cpu.threads‚17 network.sent‚ 3.921022e+06 disk.nvme1c1n1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0* proc.memory.percent‚17.231106758117676 disk.dm-1.out‚940.6865234375 network.recv‚392587& disk.nvme1c1n1.out‚4444.9951171875$ disk.nvme1n1.out‚4444.9951171875 disk.md0.in‚0 disk.md0.out‚38.28125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚8656.74072265625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚5233.87744140625% disk.nvme0n1.out‚5233.87744140625j –Å ‚(:… ÝõÒÛê’, gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚114.831' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.277 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>²i_2Æ - _wandbruntime‚1217  _runtime‚1217.ïŠuÈÇ Â ' lr-AdamW/pg1‚0.00010089915804973149 ' lr-AdamW/pg2‚0.00010089915804973149 trainer/global_step‚3099 % _timestamp‚1.7844981437260306e+09 _runtime‚ 1217.63696515 _step‚247ËxCqÍÈ Ç _step‚247 ' lr-AdamW/pg1‚0.00010089915804973149 ' lr-AdamW/pg2‚0.00010089915804973149 trainer/global_step‚3099 % _timestamp‚1.7844981437260306e+09 _runtime‚ 1217.63696515÷Ûää2É - _wandbruntime‚1217  _runtime‚1217ÚÞ®#Ê ÌO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2158882799302 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31307642 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2229155278336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31307642 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118763130391552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7844981437268627e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2165673651200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113447792620544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115676736290304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113447767716864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31297499 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21487565 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118763130391552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2165761673216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21487830 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23411493 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117218124939408 _runtime‚1217.637648411 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2165673651200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2228968573440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54708990 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13583424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7904403 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120928804042752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118763147430912 GADeviceSta-Äv\ÒtsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120928804042752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2165761673216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119377110926544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376Y§×'Ë ÑO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118763147430912 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23411491 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31297499 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21487565 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118763130391552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2165761673216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21487830 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23411493 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117218124939408 _runtime‚1217.637648411 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2165673651200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2228968573440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54708990 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13583424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7904403 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120928804042752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118763147430912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120928804042752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2165761673216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119377110926544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2158882799302 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31307642 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2229155278336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31307642 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118763130391552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7844981437268627e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2165673651200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113447792620544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115676736290304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113447767716864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54736368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13587714 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2229665916928 trainer/global_step‚3099 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2166351760896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2159558750746 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119414555757914 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23418812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31307397 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2166439790080 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120966848519680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21494326 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2159644899332 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2166351760896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2229848420352 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54726209 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23418812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115713336198144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _step‚249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115713128791040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7906859 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54726209 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13587717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54736368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120966953588224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118800496758784 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118800496758784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113483462874112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119414658945860 _runtime‚1217.892580629 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23418814 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844981439814687e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31307397 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113483487777792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117254997007168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120966953588224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118800513798144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184ï8êÞá&Î ÛM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54736368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13587714 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2229665916928 trainer/global_step‚3099 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2166351760896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2159558750746 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119414555757914 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23418812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31307397 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2166439790080 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120966848519680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21494326 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2159644899332 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2166351760896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2229848420352 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54726209 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23418812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115713336198144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _step‚249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115713128791040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7906859 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54726209 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13587717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54736368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120966953588224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118800496758784 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118800496758784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113483462874112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119414658945860 _runtime‚1217.892580629 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23418814 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844981439814687e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31307397 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113483487777792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117254997007168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120966953588224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118800513798144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184ù3ðÖq2Ï - _wandbruntime‚1217  _runtime‚1217Ψ²RÐ û 2loss/SRqqt2/assignment_loss‚0.8199061494677475 0loss/FBt1/assignment_loss‚0.3804098170020983 1loss/FBt2/assignment_loss‚0.15341464737416202 1loss/SRqqt1/assignment_loss‚1.005116633715591 /loss/FBt2/detection_loss‚1.5347101460875106 trainer/global_step‚3099 _step‚250 epoch‚3 /loss/FRt1/detection_loss‚0.5857531378680321 .loss/FRt2/detection_loss‚1.280704926859471 1loss/SRqqt1/detection_loss‚1.4766076556631123 .loss/FBt1/detection_loss‚1.190465014837014 %loss/total_loss‚13.75199661624541 $ _timestamp‚1.784498143983072e+09 _runtime‚1217.897283148 /loss/FRt1/assignment_loss‚3.155015054464904 0loss/FRt2/assignment_loss‚1.4518538555130363 1loss/SRqqt2/detection_loss‚0.7180395773927317r ÖÔýÑ € 1loss/SRqqt2/detection_loss‚0.7180395773927317 2loss/SRqqt2/assignment_loss‚0.8199061494677475 0loss/FBt1/assignment_loss‚0.3804098170020983 1loss/FBt2/assignment_loss‚0.15341464737416202 1loss/SRqqt1/assignment_loss‚1.005116633715591 /loss/FBt2/detection_loss‚1.5347101460875106 trainer/global_step‚3099 _step‚250 epoch‚3 /loss/FRt1/detection_loss‚0.5857531378680321 .loss/FRt2/detection_loss‚1.280704926859471 1loss/SRqqt1/detection_loss‚1.4766076559jÄG 6631123 .loss/FBt1/detection_loss‚1.190465014837014 %loss/total_loss‚13.75199661624541 $ _timestamp‚1.784498143983072e+09 _runtime‚1217.897283148 /loss/FRt1/assignment_loss‚3.155015054464904 0loss/FRt2/assignment_loss‚1.4518538555130363úÃ>W=Ò ‚(:² ìõÒ­øì, disk.nvme0n1.in‚0% disk.nvme1n1.out‚4447.33251953125 disk.md0.in‚0 proc.cpu.threads‚17 network.sent‚ 3.923986e+06 disk.rbd0.in‚0' proc.memory.availableMB‚ 66.05078125) disk./.usagePercent‚75.92909576411047% disk./.usageGB‚1165.7014427185059 disk.dm-1.in‚0% disk.nvme0n1.out‚5428.39697265625 disk.nvme1c1n1.in‚0 disk.md0.out‚ 38.58203125 cpu‚26.15267857336771 network.recv‚395772! disk.dm-0.out‚8851.1025390625 disk.rbd0.out‚0' disk.nvme1c1n1.out‚4447.33251953125 disk.nvme1n1.in‚0$ memory_percent‚99.59685802459717# proc.memory.rssMB‚ 2823.14453125* proc.memory.percent‚17.231106758117676 disk.dm-0.in‚0 disk.dm-1.out‚942.5146484375 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚5428.39697265625øsäšÓ ‚(: ìõÒÿö“- gpu.0.gpu‚61.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.868' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.95599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ^…Æ2Ô - _wandbruntime‚1236  _runtime‚1236èñ¬ÈÕ Â ' lr-AdamW/pg1‚0.00010086151435757259 ' lr-AdamW/pg2‚0.00010086151435757259 trainer/global_step‚3149 $ _timestamp‚1.784498162459425e+09 _runtime‚1236.370369835 _step‚251¤Î4ÍÖ Ç trainer/global_step‚3149 $ _timestamp‚1.784498162459425e+09 _runtime‚1236.370369835 _step‚251 ' lr-AdamW/pg1‚0.00010086151435757259 ' lr-AdamW/pg2‚0.00010086151435757259û)Œ2× - _wandbruntime‚1236  _runtime‚1236PuFÒ'Ø ÌO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31804394 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120631448753152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2199611393024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13797927 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚3149 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2199699415040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚252 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21826085 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚1236.371320497 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121254543789952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122831165207552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2199611393024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120631448753152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8027919 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2264114181632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119061745366768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122831060146176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23777541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115232525579264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119061728327408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 % _timestamp‚1.7844981624603221e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55571792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120631465792512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31794251 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121254440602118 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2192712274710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115232550482944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31794251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2199699415040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23777543 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23777541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117496664664576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120631465792512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8028158 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13797924 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2263940059648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122831165207552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31804394 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152§­×'Ù ÑO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122831165207552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2199611393024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120631448753152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8027919 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2264114181632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119061745366768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122831060146176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23777541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115232525579264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119061728327408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 % _timestamp‚1.7844981624603221e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55571792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120631465792512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31794251 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121254440602118 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2192712274710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115232550482944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31794251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2199699415040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23777543 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23777541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117496664664576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120631465792512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8028158 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13797924 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2263940059648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122831165207552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31804394 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31804394 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120631448753152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2199611393024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13797927 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚3149 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2199699415040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚252 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21826085 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚1236.371320497 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121254543789952üýÝPò2Ú - _wandbruntime‚1236  _runtime‚1236Æ1KËÜ&Û ÖM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚3149 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122869104623104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13802214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55589011 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2264631025664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31814306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115268245640192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31804149 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120668815120384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122869209691648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117533065460736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200289502720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23784862 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55599170 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23784864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200289502720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120668832159744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55599170 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264819820544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2193388226154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120668832159744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23784864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55589011 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2193474374740 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844981627181442e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23784862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122869209691648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200377531904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120668815120384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121291988621322 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21832600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119098600395168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117532851762176æ÷Ü ÛM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_poo.u ¯Å%l.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200289502720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23784862 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55599170 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23784864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200289502720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120668832159744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55599170 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264819820544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2193388226154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120668832159744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23784864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55589011 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2193474374740 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844981627181442e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23784862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122869209691648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200377531904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120668815120384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121291988621322 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21832600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119098600395168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117532851762176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚3149 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122869104623104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13802214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55589011 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2264631025664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31814306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115268245640192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31804149 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120668815120384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122869209691648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117533065460736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184ý:è2Ý - _wandbruntime‚1236  _runtime‚1236S€µ^Þ € % _timestamp‚1.7844981627190573e+09 0loss/FRt1/assignment_loss‚3.1552529458677294 1loss/FBt1/assignment_loss‚0.27855202644648214 /loss/FBt2/detection_loss‚1.3874095896234004 epoch‚3 _runtime‚1236.629750716 _step‚254 2loss/SRqqt2/assignment_loss‚0.8072208249768013 trainer/global_step‚3149 2loss/SRqqt1/assignment_loss‚0.8780733265205106 /loss/FRt2/detection_loss‚1.3916050668090427 1loss/SRqqt1/detection_loss‚3.3511271796931807 0loss/FRt2/assignment_loss‚1.5614640898358039 1loss/FBt2/assignment_loss‚0.06037579036523206 /loss/FRt1/detection_loss‚0.6002319805464644 0loss/SRqqt2/detection_loss‚0.768455156533666 .loss/FBt1/detection_loss‚1.316439031185531 &loss/total_loss‚15.556207008403844¢ÜÆ ß … epoch‚3 _runtime‚1236.629750716 _step‚254 2loss/SRqqt2/assignment_loss‚0.8072208249768013 trainer/global_step‚3149 2loss/SRqqt1/assignment_loss‚0.8780733265205106 /loss/FRt2/detection_loss‚1.3916050668090427 1loss/SRqqt1/detection_loss‚3.3511271796931807 0loss/FRt2/assignment_loss‚1.5614640898358039 1loss/FBt2/assignment_loss‚0.06037579036523206 /loss/FRt1/detection_loss‚0.6002319805464644 0loss/SRqqt2/detection_loss‚0.768455156533666 .loss/FBt1/detection_loss‚1.316439031185531 &loss/total_loss‚15.556207008403844 % _timestamp‚1.7844981627190573e+09 0loss/FRt1/assignment_loss‚3.1552529458677294 1loss/FBt1/assignment_loss‚0.27855202644648214 /loss/FBt2/detection_loss‚1.3874095896234004þê:rŸà ‚(:… ûõÒ¦ªµ+ gpu.0.gpu‚84.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚144.273' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚48.091 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êãóö2á ‚(:§ ûõÒºñÌ+ network.recv‚400252) disk./.usagePercent‚75.96933165778363 disk.nvme0n1.in‚0# disk.nvme1n1.out‚4447.939453125 disk.dm-1.out‚ 942.662109375 disk.rbd0.out‚0 disk.md0.out‚38.9375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚7150.5751953125& proc.memory.availableMB‚ 65.7421875$ disk./.usageGB‚1166.319164276123$ disk.nvme0n1.out‚7150.5751953125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚99.59874153137207* proc.memory.percent‚17.231106758117676 cpu‚26.150117947733943 network.sent‚ 3.986286e+06 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚4447.939453125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2823.14453125 proc.cpu.threads‚17 disk.dm-0.in‚0" disk.dm-0.out‚10572.9697265625lÁ·2â - _wandbruntime‚1255  _runtime‚1255)äÈã  _runtime‚1255.194614851 _step‚255 ' lr-AdamW/pg1‚0.00010082307008941377 ' lr-AdamW/pg2‚0.00010082307008941377 trainer/global_step‚3199 $ _timestamp‚1.784498181283673e+09e·@lÍä Ç _runtime‚1255.194614851 _step‚255 ' lr-AdamW/pg1‚0.00010082307008941377 ' lr-AdamW/pg2‚0.00010082307008941377 trainer/global_step‚3199 $ _timestamp‚1.784498181283673e+09ÿî­«­2å - _wandbruntime‚1255  _runtime‚1255ËquLÒ'æ ÌO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56434594 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2298927634944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22163242 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _runtime‚1255.195605103 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117017308345344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚3199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122499767114752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14012424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2226627898592 _step‚256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56444739 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8150815 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124733316249600 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22162982 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119316211076608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123131873465526 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2226541750118 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122499784154112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32301146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124733421310976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24143591 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123131976653360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119316418490880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2233549134848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120905348754768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32291003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844981812846506e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117017283441664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120905331715408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2299110145536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122499767114752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8150558 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24143593 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124733316249600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248ì>ïŽ%ç ÑO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32291003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844981812846506e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117017283441664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120905331715408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2299110145536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122499767114752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8150558 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24143593 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124733316249600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56434594 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2298927634944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22163242 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _runtime‚1255.195605103 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117017308345344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚3199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122499767114752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14012424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2226627898592 _step‚256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56444739 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8150815 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124733316249600 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22162982 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119316211076608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123131873465526 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2226541750118 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122499784154112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32301146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124733421310976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24143591 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123131976653360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_£‡åÀpool.allocated‚330 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119316418490880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2233549134848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120905348754768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104€lYçÓ2è - _wandbruntime‚1255  _runtime‚1255ê¸ýÜ&é ÖM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123169421484730 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56461972 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚124771360726528 % _timestamp‚1.7844981815429142e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56451813 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8153232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123169524672676 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24150914 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚1255.453980458 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24150912 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117053003502592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚122537150521344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56451813 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2299806720000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124771465795072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8152997 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2299630508032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2234227244544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117052978598912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32311058 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22169949 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119352810222592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122537133481984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22169711 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122537133481984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32300901 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56461972 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119352609106944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124771465795072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32311058 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚3199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120942220822528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2234315273728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0½Ø*Þá&ê ÛM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2234227244544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32300901 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24150914 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14016717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚122537150521344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56451813 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2299806720000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124771465795072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8152997 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2299630508032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2234227244544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117052978598912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32311058 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22169949 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119352810222592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122537133481984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22169711 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122537133481984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32300901 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56461972 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119352609106944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124771465795072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32311058 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚3199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120942220822528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2234315273728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123169421484730 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56461972 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚124771360726528 % _timestamp‚1.7844981815429142e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56451813 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8153232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123169524672676 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24150914 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚1255.453980458 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24150912 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117053003502592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1495765745639801 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21789519488811493 'particle/f_score‚0.6133667826652527 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6320371031761169 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9588620066642761 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.8901552557945251 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6243666410446167 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/sensitivity‚0.5938065648078918 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13942797482013702 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05336131900548935 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6124156713485718 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.40135064721107483 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9938715696334839 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5502758622169495 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.666532039642334 +particle/specificity‚0.9063980579376221 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3499999940395355 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5192307829856873 trainer/global_step‚3203 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.612083911895752 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18175828456878662 *jet/accuracy_1_of_3‚0.9122198224067688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9769558906555176 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5634536147117615 particle/accuracy_1_of_3‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.762499988079071 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6796075105667114 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15162082016468048 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7466126084327698 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.68160480260849 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15291550755500793 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9906201362609863 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15226219594478607 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2430555671453476 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5897427201271057 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7904959321022034 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 % _timestamp‚1.7844981981353908e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17121034860610962 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19759096205234528 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9908549785614014 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8163474202156067 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.151650071144104 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.053866397589445114 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9775203466415405 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005499241407960653 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9407894611358643 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.991671085357666 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.323908269405365 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.139654278755188 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7099430561065674 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27793461084365845 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1649191826581955 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.672510027885437 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9895210266113281 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042141254991292953 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13802051544189453 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6841838955879211 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5870640873908997 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 -jet/accuracy_4_of_4‚0.0023809524718672037 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44003796577453613 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38804158568382263 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15445315837860107 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7188595533370972 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4771823287010193 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7527261972427368 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6283144950866699 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.369682252407074 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5644736289978027 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5357142686843872 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7278094291687012 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6836695075035095 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.76958829164505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5357142686843872 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43325600028038025 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6295180320739746 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.9302437901496887 particle/accuracy_1_of_1‚1 _step‚259 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11974384635686874 ( Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9901490807533264 *jet/accuracy_1_of_1‚0.2992931008338928 epoch‚3 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3727444112300873 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3760041892528534 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6381704211235046 jet/accuracy_4_of_5‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6474422812461853 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21274134516716003 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7904959321022034 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876543283462524 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8600676655769348 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ p=ó¡ï0.02294921875 +jet/accuracy_2_of_2‚0.12375646829605103 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.05472215637564659 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9906201362609863 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.618704080581665 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953703880310059 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 _runtime‚ 1272.04675158 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9503694176673889 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5434355735778809 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37717169523239136 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1482517421245575 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9832178950309753 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010358504019677639 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009245849214494228 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.134685680270195 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0465664342045784 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9294871687889099 particle/accuracy_2_of_3‚1 'particle/accuracy‚0.839277982711792 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42500001192092896 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7230768799781799 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9933944940567017 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1639680117368698 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4915563464164734 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15445315837860107 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.054219264537096024 6validation_average_jet_accuracy‚0.3445129990577698 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6880592703819275 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.7803105115890503 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6381704211235046 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11926867067813873 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0465664342045784 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.743219792842865 (jet/accuracy_2_of_4‚0.74765545129776 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7011904716491699 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9645298719406128 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9066239595413208 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7696415781974792 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.4453125 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7310060858726501 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5975097417831421 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.49629610776901245 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9887769818305969 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9672352075576782 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.8951146006584167 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.166129931807518 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.055263668298721313 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9679428935050964 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6428571343421936 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7671704888343811 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.034139737486839294 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7151347994804382 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7018796801567078 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6044874787330627 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09739583730697632 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24458642303943634 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.133435919880867 .particle/accuracy_3_of_3‚0.999514102935791 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17444083094596863 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12825921177864075 *jet/accuracy_1_of_2‚0.7794430255889893 jet/accuracy_5_of_5‚0 )loss/val_total_loss‚12.84495813021508 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10838185995817184 .particle/accuracy_4_of_4‚0.940419614315033 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6350304484367371 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7702686786651611 jet/accuracy_2_of_5‚1 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 /particle/accuracy_3_of_4‚0.9933863282203674 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37121209502220154 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018444575369358063 *jet/accuracy_2_of_3‚0.5005638003349304 +jet/accuracy_3_of_4‚0.06067676097154617 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9597857594490051 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8742474317550659 ("Purity/2FRt2SRqqt0FBt/event_purity‚0õܱ±=ò «{ particle/accuracy_2_of_3‚1 'particle/accuracy‚0.839277982711792 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42500001192092896 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7230768799781799 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9933944940567017 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1639680117368698 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4915563464164734 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15445315837860107 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.054219264537096024 6validation_average_jet_accuracy‚0.3445129990577698 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6880592703819275 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.7803105115890503 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6381704211235046 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11926867067813873 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0465664342045784 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.743219792842865 (jet/accuracy_2_of_4‚0.74765545129776 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7011904716491699 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9645298719406128 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9066239595413208 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7696415781974792 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.4453125 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7310060858726501 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5975097417831421 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.49629610776901245 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9887769818305969 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9672352075576782 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.8951146006584167 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.166129931807518 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.055263668298721313 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9679428935050964 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6428571343421936 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7671704888343811 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.034139737486839294 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7151347994804382 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7018796801567078 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6044874787330627 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09739583730697632 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24458642303943634 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.133435919880867 .particle/accuracy_3_of_3‚0.999514102935791 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17444083094596863 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12825921177864075 *jet/accuracy_1_of_2‚0.7794430255889893 jet/accuracy_5_of_5‚0 )loss/val_total_loss‚12.84495813021508 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10838185995817184 .particle/accuracy_4_of_4‚0.940419614315033 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6350304484367371 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7702686786651611 jet/accuracy_2_of_5‚1 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 /particle/accuracy_3_of_4‚0.9933863282203674 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37121209502220154 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018444575369358063 *jet/accuracy_2_of_3‚0.5005638003349304 +jet/accuracy_3_of_4‚0.06067676097154617 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9597857594490051 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8742474317550659 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3499999940395355 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6791110634803772 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7378177046775818 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12345097959041595 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1495765745639801 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21789519488811493 'particle/f_score‚0.6133667826652527 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6320371031761169 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9588620066642761 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.8901552557945251 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6243666410446167 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/sensitivity‚0.5938065648078918 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13942797482013702 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05336131900548935 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6124156713485718 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.40135064721107483 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9938715696334839 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5502758622169495 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.666532039642334 +particle/specificity‚0.9063980579376221 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3499999940395355 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5192307829856873 trainer/global_step‚3203 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.612083911895752 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18175828456878662 *jet/accuracy_1_of_3‚0.9122198224067688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9769558906555176 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5634536147117615 particle/accuracy_1_of_3‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.762499988079071 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6796075105667114 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15162082016468048 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7466126084327698 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.68160480260849 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15291550755500793 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9906201362609863 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15226219594478607 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2430555671453476 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5897427201271057 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7904959321022034 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 % _timestamp‚1.7844981981353908e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17121034860610962 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19759096205234528 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9908549785614014 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8163474202156067 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.151650071144104 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.053866397589445114 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9775203466415405 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005499241407960653 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9407894611358643 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.991671085357666 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.323908269405365 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.139654278755188 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7099430561065674 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27793461084365845 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1649191826581955 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.672510027885437 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9895210266113281 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042141254991292953 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13802051544189453 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6841838955879211 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5870640873908997 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 -jet/accuracy_4_of_4‚0.0023809524718672037 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44003796577453613 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38804158568382263 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15445315837860107 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7188595533370972 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4771823287010193 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7527261972427368 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6283144950866699 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.369682252407074 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5644736289978027 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5357142686843872 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7278094291687012 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6836695075035095 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.76958829164505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5357142686843872 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43325600028038025 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6295180320739746 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.9302437901496887 particle/accuracy_1_of_1‚1 _step‚259 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11974384635686874 ( Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9901490807533264 *jet/accuracy_1_of_1‚0.2992931008338928 epoch‚3 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3727444112300873 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3760041892528534 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6381704211235046 jet/accuracy_4_of_5‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6474422812461853 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21274134516716003 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7904959321022034 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876543283462524 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8600676655769348 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_2_of_2‚0.12375646829605103 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.05472215637564659 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9906201362609863 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.618704080581665 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953703880310059 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 _runtime‚ 1272.04675158 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9503694176673889 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5434355735778809 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37717169523239136 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1482517421245575 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9832178950309753 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010358504019677639 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009245849214494228 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.134685680270195 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0465664342045784 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9294871687889099ƒÙÅ=ó ‚(:² ™õÒØèš+# proc.memory.rssMB‚ 2822.41796875 cpu‚26.182350398563678 network.sent‚ 4.110756e+06 network.recv‚409305) disk./.usagePercent‚75.96632263750115 disk.md0.in‚0 disk.md0.out‚ 39.48046875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚4642.19970703125 disk.dm-1.in‚0# memory_percent‚98.7952470779419( proc.memory.availableMB‚ 197.38671875* proc.memory.percent‚17.226672172546387& disk.nvme0c0n1.out‚7891.7998046875$ disk.nvme0n1.out‚7891.7998046875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1166.2729682922363 disk.rbd0.in‚0% disk.nvme1n1.out‚4642.19970703125 disk.dm-0.in‚0# disk.dm-0.out‚11443.15087890625 disk.dm-1.out‚ 1006.78515625ÞkèOšô ‚(: ™õÒéú¸+ gpu.0.gpu‚71.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚182.422' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.80733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mðëò2õ - _wandbruntime‚1289  _runtime‚1289ªÆJÈö  ' lr-AdamW/pg2‚0.00010078382586101803 trainer/global_step‚3249 $ _timestamp‚1.784498215868064e+09 _runtime‚1289.779020909 _step‚260 ' lr-AdamW/pg1‚0.000100783825861018030Éõ³Í÷ Ç ' lr-AdamW/pg1‚0.00010078382586101803 ' lr-AdamW/pg2‚0.00010078382586101803 trainer/global_step‚3249 $ _timestamp‚1.784498215868064e+09 _runtime‚1289.779020909 _step‚260„ϧŒ2ø - _wandbruntime‚1289  _runtime‚1289¢å$Q!ù ÌO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123717441611120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125998189804852 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚3249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125340749071360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22674200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123717424571760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119727489471488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122082098930688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32965164 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8352940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57712275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14321004 _step‚261 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125340766110720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32955021 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122081895710720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127628761326592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2354609459200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127628656265216 ZHDeviceStatsMonitor.on_train_batch_start­Š/requested_bytes.small_pool.freed‚ 2280765233092 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32955021 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32965164 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127628656265216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2287995215872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 % _timestamp‚1.7844982158689237e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57702130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125998292992686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚1289.783401827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14321007 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152³ÖT×'ú ÑO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2287907193856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57712275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14321004 _step‚261 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125340766110720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32955021 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122081895710720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127628761326592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2354609459200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127628656265216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2280765233092 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32955021 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32965164 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127628656265216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2287995215872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 % _timestamp‚1.7844982158689237e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57702130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125998292992686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚1289.783401827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14321007 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123717441611120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125998189804852 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚3249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125340749071360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22674200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123717424571760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119727489471488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122082098930688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32965164 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8352940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24754432 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚3249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8355580 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22680877 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123754313678880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125378132477952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125378132477952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24754430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2288585303552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57729508 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57719349 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2288673332736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32975076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2355128932864 # _timestamp‚1.78449821612746e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57719349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24754430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122118493967872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127666700742144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125378115438592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32964919 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127666805810688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24754432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125378115438592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8355344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1290.038487639 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119763184628736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22680638 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946ÔFx¦%ÿ ÙM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22680638 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2288673332736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14325294 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24754432 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚3249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8355580 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22680877 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123754313678880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125378132477952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125378132477952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24754430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2288585303552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57729508 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57719349 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2288673332736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32975076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2355128932864 # _timestamp‚1.78449821612746e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57719349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24754430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122118493967872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127666700742144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125378115438592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32964919 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127666805810688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24754432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125378115438592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8355344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1290.038487639 PBDeviceStatsMonitor.on_trai‚n&9n_batch_end/active_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119763184628736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184†»)ç2€ - _wandbruntime‚1290  _runtime‚12907ˆ*f ý 1loss/SRqqt2/detection_loss‚0.6840582290211654 _step‚263 /loss/FRt1/assignment_loss‚3.042185974067153 2loss/SRqqt1/assignment_loss‚0.9716433843016519 1loss/SRqqt1/detection_loss‚1.5124253492458914 trainer/global_step‚3249 % _timestamp‚1.7844982161283782e+09 1loss/FBt1/assignment_loss‚0.27726179065822903 .loss/FBt1/detection_loss‚5.618960668369532 epoch‚4 0loss/FRt2/assignment_loss‚1.3855226438492536 2loss/SRqqt2/assignment_loss‚0.8353087250970607 -loss/FRt1/detection_loss‚0.58627346730392 /loss/FRt2/detection_loss‚1.3510308433134923 .loss/FBt2/detection_loss‚1.482098504456558 &loss/total_loss‚17.918032652065023 _runtime‚1290.042363976 1loss/FBt2/assignment_loss‚0.17126307238111418úöt ‚ ‚ /loss/FRt1/assignment_loss‚3.042185974067153 2loss/SRqqt1/assignment_loss‚0.9716433843016519 1loss/SRqqt1/detection_loss‚1.5124253492458914 trainer/global_step‚3249 % _timestamp‚1.7844982161283782e+09 1loss/FBt1/assignment_loss‚0.27726179065822903 .loss/FBt1/detection_loss‚5.618960668369532 epoch‚4 0loss/FRt2/assignment_loss‚1.3855226438492536 2loss/SRqqt2/assignment_loss‚0.8353087250970607 -loss/FRt1/detection_loss‚0.58627346730392 /loss/FRt2/detection_loss‚1.3510308433134923 .loss/FBt2/detection_loss‚1.482098504456558 &loss/total_loss‚17.918032652065023 _runtime‚1290.042363976 1loss/FBt2/assignment_loss‚0.17126307238111418 1loss/SRqqt2/detection_loss‚0.6840582290211654 _step‚263‡ž.Ê@ƒ ‚(:µ ·õÒ£Á+ disk.md0.in‚0 disk.md0.out‚ 39.96484375% disk.nvme0n1.out‚7898.11865234375* proc.memory.percent‚17.226672172546387 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ memory_percent‚98.79612922668457 proc.cpu.threads‚17 network.recv‚418186% disk.nvme1n1.out‚4707.78564453125# disk.dm-0.out‚11449.87255859375# proc.memory.rssMB‚ 2822.41796875 cpu‚26.17666957832137) disk./.usagePercent‚75.96589575914892 disk.dm-0.in‚0" disk.dm-1.out‚1070.87548828125 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚7898.11865234375' proc.memory.availableMB‚ 197.2421875 network.sent‚ 4.232025e+06$ disk./.usageGB‚1166.266414642334' disk.nvme1c1n1.out‚4707.78564453125 disk.nvme1n1.in‚0Ôƒ´š„ ‚(: ·õÒ»÷í+ gpu.0.gpu‚64.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.33366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñé‚Ö2… - _wandbruntime‚1308  _runtime‚1308ADÁ È†  _runtime‚ 1308.60937572 _step‚264 ' lr-AdamW/pg1‚0.00010074378230096134 ' lr-AdamW/pg2‚0.00010074378230096134 trainer/global_step‚3299 % _timestamp‚1.7844982346967795e+09Ð͇ Ç ' lr-AdamW/pg1‚0.00010074378230096134 ' lr-AdamW/pg2‚0.00010074378230096134 trainer/global_step‚3299 % _timestamp‚1.7844982346967795e+09 _runtime‚ 1308.60937572 _step‚264ˆsΨ2ˆ - _wandbruntime‚1308  _runtime‚1308 `2Ð'‰ ÊO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129530912368640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23010241 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2314680856974 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25113159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127209067432960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127875622668260 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127209084472320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784498234697401e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25113161 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123901849521152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2314594708500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58564932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127209084472320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125561027959760 trainer/global_step‚3299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2321932957696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125561044999120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2389602187264 _step‚265 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129531017430016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25113161 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123901640009728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127875725856094 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2321932957696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14535504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127209067432960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33461916 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121512247333888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58575077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23010496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14535507 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33451773 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121512222430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127209067432960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33461916 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121512247333888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58575077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23010496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14535507 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33451773 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121512222430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129530912368640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23010241 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2314680856974 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25113159 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127209067432960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127875622668260 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127209084472320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784498234697401e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25113161 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123901849521152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2314594708500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58564932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127209084472320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125561027959760 trainer/global_step‚3299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2321932957696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125561044999120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2389602187264 _step‚265 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129531017430016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25113161 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123901640009728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127875725856094 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2321932957696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14535504‰jWâ2‹ - _wandbruntime‚1308  _runtime‚1308ݶÕK~!Œ ÕM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33471828 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.784498234955686e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25120482 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58592310 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315270659944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14539794 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123938242432512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25120480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127913273875410 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2390113243648 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2322611074560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2315356808530 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚1308.867062199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129568956845568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8477214 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2390299941376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33471828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125597900027520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322523045376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125597917066880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322523045376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121547942491136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127246450839552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23017008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBÐu…Û]DeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚3299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129569061914112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25120480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _step‚266 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127246450839552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129569061914112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23017272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14539797 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58592310 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2322611074560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8477475Ö†d´à& ÚM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33471828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125597900027520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322523045376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125597917066880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322523045376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121547942491136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127246450839552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23017008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚3299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129569061914112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25120480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _step‚266 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127246450839552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129569061914112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23017272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14539797 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58592310 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2322611074560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8477475 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33471828 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.784498234955686e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25120482 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58592310 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315270659944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14539794 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123938242432512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25120480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127913273875410 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2390113243648 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2322611074560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2315356808530 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚1308.867062199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129568956845568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8477214 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2390299941376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Š›óü=2Ž - _wandbruntime‚1308  _runtime‚1308wã3á  /loss/FRt2/detection_loss‚1.4274502521163899 &loss/total_loss‚12.600831309389129 trainer/global_step‚3299 % _timestamp‚1.7844982349566233e+09 _step‚267 0loss/FRt2/assignment_loss‚1.2313872030744815 2loss/SRqqt1/assignment_loss‚0.8462714502574319 1loss/FBt1/assignment_loss‚0.23403519618757077 /loss/FRt1/detection_loss‚0.6135575946983193 1loss/SRqqt2/detection_loss‚0.8138044426638221 epoch‚4 _runtime‚1308.868630422 0loss/FRt1/assignment_loss‚2.9048017164516877 2loss/SRqqt2/assignment_loss‚0.7492528212452964 1loss/FBt2/assignment_loss‚0.02595010541654119 0loss/SRqqt1/detection_loss‚1.470913569449301 /loss/FBt2/detection_loss‚0.9569245119328116 /loss/FBt1/detection_loss‚1.3264824458954745ƒ5®  † 0loss/FRt1/assignment_loss‚2.9048017164516877 2loss/SRqqt2/assignment_loss‚0.7492528212452964 1loss/FBt2/assignment_loss‚0.02595010541654119 0loss/SRqqt1/detection_loss‚1.470913569449301 /loss/FBt2/detection_loss‚0.9569245119328116 /loss/FBt1/detection_loss‚1.3264824458954745 /loss/FRt2/detection_loss‚1.4274502521163899 &loss/total_loss‚12.600831309389129 trainer/global_step‚3299 % _timestamp‚1.7844982349566233e+09 _step‚267 0loss/FRt2/assignment_loss‚1.2313872030744815 2loss/SRqqt1/assignment_loss‚0.8462714502574319 1loss/FBt1/assignment_loss‚0.23403519618757077 /loss/FRt1/detection_loss‚0.6135575946983193 1loss/SRqqt2/detection_loss‚0.8138044426638221 epoch‚4 _runtime‚1308.868630422‹`ªÁ@‘ ‚(:µ ÆõÒ„¸Ñ+) disk./.usagePercent‚75.96588408087038% disk./.usageGB‚1166.2662353515625% disk.nvme1n1.out‚4709.28466796875 disk.dm-1.in‚0" disk.dm-1.out‚1071.04736328125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚4709.28466796875 network.sent‚ 4.284043e+06 disk.md0.out‚ 40.3046875# disk.dm-0.out‚11454.87841796875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0$ memory_percent‚98.79636764526367# proc.memory.rssMB‚ 2822.41796875* proc.memory.percent‚17.226672172546387 disk.md0.in‚0 disk.dm-0.in‚0' disk.nvme0c0n1.out‚7902.51416015625% disk.nvme0n1.out‚7902.51416015625& proc.memory.availableMB‚ 197.203125 network.recv‚422293 disk.nvme1n1.in‚0 disk.nvme1c1n1.in‚0 cpu‚26.174113954523317 proc.cpu.threads‚17 ïq·›’ ‚(: ÆõÒ•®ÿ+ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚159.121' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚53.040333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ÕÝá2“ - _wandbruntime‚1327  _runtime‚1327À.¸É” à ' lr-AdamW/pg1‚0.00010070294005062259 ' lr-AdamW/pg2‚0.00010070294005062259 trainer/global_step‚3349 % _timestamp‚1.7844982532166302e+09 _runtime‚1327.129393086 _step‚268”kÇóΕ È _runtime‚1327.129393086 _step‚268 ' lr-AdamW/pg1‚0.00010070294005062259 ' lr-AdamW/pg2‚0.00010070294005062259 trainer/global_step‚3349 % _timestamp‚1.7844982532166302e+09Œ·Lv>2– - _wandbruntime‚1327  _runtime‚1327Æ”Ò'— ÌO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25479211 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2355870699520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25479211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2348424183908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1327.129491076 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129077402833920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 % _timestamp‚1.7844982532171915e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131433273533440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59427734 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25479209 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2355782677504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123297005196288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129753055531668 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597833 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14750004 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2355782677504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33958668 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚269 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127404648387120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14750007 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131433168472064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129077385794560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25479209 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129077402833920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2355870699520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127404631347760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33948525 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131433168472064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123296980292608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33958668 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚269 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127404648387120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14750007 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131433168472064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129077385794560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25479209 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129077402833920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2355870699520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127404631347760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33948525 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131433168472064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123296980292608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25479211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2348424183908 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1327.129491076 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129077402833920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 % _timestamp‚1.7844982532171915e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131433273533440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59427734 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25479209 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2355782677504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123297005196288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129753055531668Ràû…2™ - _wandbruntime‚1327  _runtime‚1327¥.f´Û&š ÕM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123332675449856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2425091316736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129114752161792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125757766766592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129114769201152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2349100135352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2356548816384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127441503415520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25486530 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600297 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123332700353536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2356460787200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14754294 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129114769201152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129790706738818 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59455112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59455112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25486530 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 195086336 _step‚270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23354850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59444953 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33958423 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219990016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2356548816384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129114752161792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131471318017536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33968580 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 $ _timestamp‚1.784498253788673e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131471318017536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚1327.699718805 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33968580 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600553 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59444953 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2356460787200 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14754297 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25486532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2425286403072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23354591 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131471212948992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33958423 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184®Ÿqwà&› ÚM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129114752161792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125757766766592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129114769201152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2349100135352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2356548816384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127441503415520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25486530 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600297 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123332700353536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2356460787200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14754294 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129114769201152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129790706738818 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59455112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59455112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25486530 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 195086336 _step‚270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23354850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59444953 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33958423 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219990016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2356548816384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129114752161792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131471318017536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33968580 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 $ _timestamp‚1.784498253788673e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131471318017536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚1327.699718805 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33968580 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600553 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59444953 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2356460787200 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14754297 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25486532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2425286403072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23354591 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131471212948992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33958423 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123332675449856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2425091316736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152ŽÎyPZ2œ - _wandbruntime‚1327  _runtime‚1327«aÄ ý .loss/FBt2/detection_loss‚3.246868926283129 &loss/total_loss‚15.030663296390891 epoch‚4 /loss/FRt1/assignment_loss‚2.881266684443862 0loss/FRt2/assignment_loss‚1.3263494509754452 2loss/SRqqt2/assignment_loss‚0.6795599822328275 1loss/FBt2/assignment_loss‚0.01956850343276974 .loss/FBt1/detection_loss‚1.447106351497041 trainer/global_step‚3349 _runtime‚1327.700346086 _step‚271 2loss/SRqqt1/assignment_loss‚0.9861084895019303 /loss/FRt2/detection_loss‚1.4303963481149709 0loss/SRqqt1/detection_loss‚1.491889276967413 0loss/FBt1/assignment_loss‚0.1896822751943148 /loss/FRt1/detection_loss‚0.6127352412109829 % _timestamp‚1.7844982537896714e+09 1loss/SRqqt2/detection_loss‚0.7191317665362055‘›I­ž ‚ .loss/FBt1/detection_loss‚1.447106351497041 trainer/global_step‚3349 _runtime‚1327.700346086 _step‚271 2loss/SRqqt1/assignment_loss‚0.9861084895019303 /loss/FRt2/detection_loss‚1.4303963481149709 0loss/SRqqt1/detection_loss‚1.491889276967413 0loss/FBt1/assignment_loss‚0.1896822751943148 /loss/FRt1/detection_loss‚0.6127352412109829 % _timestamp‚1.7844982537896714e+09 1loss/SRqqt2/detection_loss‚0.7191317665362055 .loss/FBt2/detection_loss‚3.246868926283129 &loss/total_loss‚15.030663296390891 epoch‚4 /loss/FRt1/assignment_loss‚2.881266684443862 0loss/FRt2/assignment_loss‚1.3263494509754452 2loss/SRqqt2/assignment_loss‚0.6795599822328275 1loss/FBt2/assignment_loss‚0.01956850343276974ã*4‘<Ÿ ‚(:± ÕõÒñÜ+ disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4710.0830078125 disk.md0.in‚0! disk.dm-1.out‚1071.2861328125 disk.nvme0c0n1.in‚0 network.sent‚ 4.346454e+06$ disk.nvme1n1.out‚4710.0830078125 disk.dm-0.in‚0# disk.dm-0.out‚11455.57373046875 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.226672172546387 cpu‚26.171616699617395% disk.nvme0n1.out‚7903.63623046875 disk.md0.out‚ 40.76953125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚7903.63623046875 proc.cpu.threads‚17% disk./.usageGB‚1166.2663459777832 disk.nvme1n1.in‚0$ memory_percent‚98.79722595214844$ proc.memory.availableMB‚197.0625# proc.memory.rssMB‚ 2822.41796875 network.recv‚426587) disk./.usagePercent‚75.96589128661671 disk.nvme0n1.in‚0º} š  ‚(: ÕõÒÓ±‰, gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.789' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.26299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jZ Ò2¡ - _wandbruntime‚1345  _runtime‚1345­¼ïæÇ¢ Á & lr-AdamW/pg1‚0.0001006612997641733 & lr-AdamW/pg2‚0.0001006612997641733 trainer/global_step‚3399 % _timestamp‚1.7844982720497928e+09 _runtime‚1345.961278694 _step‚272PyxˆÌ£ Æ & lr-AdamW/pg1‚0.0001006612997641733 & lr-AdamW/pg2‚0.0001006612997641733 trainer/global_step‚3399 % _timestamp‚1.7844982720497928e+09 _runtime‚1345.961278694 _step‚272öá 2¤ - _wandbruntime‚1345  _runtime‚13456 nK¥ ËO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127541314018304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133335424575488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131630591582910 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125081738155008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2382253659316 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133335424575488 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2389808441344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2459378934784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2389720419328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25845261 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25845259 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125081763058688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25845259 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129248251775120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2459550959616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚130945721195520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2389808441344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23685238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23685002 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8720731 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127541117089792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13J˜™f† 1630488395076 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34445277 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60290536 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133335529636864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60300681 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129248234735760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130945704156160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2382339807790 $ _timestamp‚1.784498272050417e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130945704156160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2382339807790 $ _timestamp‚1.784498272050417e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133335424575488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131630591582910 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125081738155008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2382253659316 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133335424575488 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2389808441344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2459378934784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2389720419328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25845261 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25845259 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125081763058688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25845259 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129248251775120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2459550959616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚130945721195520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2389808441344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23685238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23685002 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8720731 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127541117089792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131630488395076 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34445277‘ò¨˜q2§ - _wandbruntime‚1346  _runtime‚1346l|zÜ&¨ ÖM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131668036414280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60317914 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2460075115008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131668139602226 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34455175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130983070523392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25852580 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133373574120960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60317914 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25852582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125117433312256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125117458215936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7844982723085523e+09 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚274 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2460253424128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34465332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127577711640064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129285106803520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2390486558208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25852580 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8723175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127577508427264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1346.219636085 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23691727 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34465332 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2390398529024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712Ø's$© ÛM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60317914 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2460075115008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131668139602226 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34455175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130983070523392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25852580 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133373574120960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60317914 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25852582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125117433312256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125117458215936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7844982723085523e+09 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚274 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2460253424128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34465332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127577711640064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129285106803520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2390486558208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25852580 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8723175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127577508427264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1346.219636085 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23691727 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34465332 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2390398529024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1þ„î˜n52 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131668036414280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104’p\Ô2ª - _wandbruntime‚1346  _runtime‚13467-2B« € /loss/FRt2/detection_loss‚1.5140888681207263 0loss/SRqqt1/detection_loss‚1.499656110918041 /loss/FBt2/detection_loss‚1.3577163691503653 &loss/total_loss‚14.551502112797094 epoch‚4 trainer/global_step‚3399 _runtime‚1346.220150976 0loss/FRt2/assignment_loss‚1.4232573347056614 2loss/SRqqt2/assignment_loss‚0.7659880673670942 1loss/FBt1/assignment_loss‚0.17744854631934573 1loss/FBt2/assignment_loss‚0.10473158437150928 /loss/FRt1/detection_loss‚0.6102156736970079 _step‚275 2loss/SRqqt1/assignment_loss‚0.7527531962697358 1loss/SRqqt2/detection_loss‚0.7011856612466764 .loss/FBt1/detection_loss‚2.725118062167894 % _timestamp‚1.7844982723094935e+09 0loss/FRt1/assignment_loss‚2.91934263846303654¦Ù7 ¬ … 2loss/SRqqt1/assignment_loss‚0.7527531962697358 1loss/SRqqt2/detection_loss‚0.7011856612466764 .loss/FBt1/detection_loss‚2.725118062167894 % _timestamp‚1.7844982723094935e+09 0loss/FRt1/assignment_loss‚2.9193426384630365 /loss/FRt2/detection_loss‚1.5140888681207263 0loss/SRqqt1/detection_loss‚1.499656110918041 /loss/FBt2/detection_loss‚1.3577163691503653 &loss/total_loss‚14.551502112797094 epoch‚4 trainer/global_step‚3399 _runtime‚1346.220150976 0loss/FRt2/assignment_loss‚1.4232573347056614 2loss/SRqqt2/assignment_loss‚0.7659880673670942 1loss/FBt1/assignment_loss‚0.17744854631934573 1loss/FBt2/assignment_loss‚0.10473158437150928 /loss/FRt1/detection_loss‚0.6102156736970079 _step‚275“°(?­ ‚(:´ äõÒݽÆ+ disk.nvme1n1.in‚0 disk.md0.out‚ 40.93359375( proc.memory.availableMB‚ 196.99609375# proc.memory.rssMB‚ 2822.41796875 cpu‚26.16918491883681# disk.dm-0.out‚11457.21923828125! disk.dm-1.out‚1071.3095703125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4710.5576171875 network.sent‚ 4.408634e+06 network.recv‚430806) disk./.usagePercent‚75.96589004424666 disk.dm-0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚7905.21728515625 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚4710.5576171875 disk.md0.in‚0$ memory_percent‚98.79763126373291* proc.memory.percent‚17.226672172546387% disk./.usageGB‚1166.2663269042969 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚7905.21728515625 õAꎮ ‚(:ƒ äõÒïó+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.738' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³±M"2¯ - _wandbruntime‚1364  _runtime‚1364-?"É° à _runtime‚1364.787377822 _step‚276 ' lr-AdamW/pg1‚0.00010061886210856721 ' lr-AdamW/pg2‚0.00010061886210856721 trainer/global_step‚3449 % _timestamp‚1.7844982908752432e+09‡g²Î± È _runtime‚1364.787377822 _step‚276 ' lr-AdamW/pg1‚0.00010061886210856721 ' lr-AdamW/pg2‚0.00010061886210856721 trainer/global_step‚3449 % _timestamp‚1.7844982908752432e+09”L™¾Ä2² - _wandbruntime‚1364  _runtime‚1364•Æq}Ò'³ ÌO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126866520921088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2416083134724 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34942029 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 trainer/global_step‚3449 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26211309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423746183168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24023851 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8844595 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7844982908758194e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚132814022517760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2423658161152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133508024446318 _runtime‚1364.787446912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34942029 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26211309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61153338 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135237680678912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2423658161152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚132814039557120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129360840135680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2494344118272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135237785740288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2494539211776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135237785740288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131091838123760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15179004 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26211311 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61163483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24023599 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34952172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2423746183168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129361060132864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133507921258484 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135237680678912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2423658161152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133508024446318 _runtime‚1364.787446912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34942029 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26211309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61153338 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135237680678912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2423658161152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚132814039557120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129360840135680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2494344118272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135237785740288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2494539211776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135237785740288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131091838123760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15179004 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26211311 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61163483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24023599 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34952172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2423746183168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129361060132864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133507921258484 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135237680678912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34942029 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 trainer/global_step‚3449 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26211309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423746183168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24023851 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8844595 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7844982908758194e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚132814022517760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135275830224384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2424336270848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34951927 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚278 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129397234102784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132851388884992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24030571 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61180716 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133545469277688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26218630 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34951927 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚132851388884992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2424336270848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135275725155840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844982911316013e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61180716 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126902216078336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2495042928128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132851405924352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424424300032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132851405924352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129397447801344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131128710191520 REDeviceStatsMonitor.on_train_batch_endMqêu /active_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26218632 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24030305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8847011 trainer/global_step‚3449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15183297 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133545572465634 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2416845234754 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Â#Ká&¹ ÛM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135275725155840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844982911316013e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61180716 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126902216078336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2495042928128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132851405924352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424424300032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132851405924352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129397447801344 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131128710191520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26218632 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24030305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8847011 trainer/global_step‚3449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15183297 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133545572465634 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2416845234754 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135275830224384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2424336270848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34951927 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚278 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129397234102784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132851388884992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24030571 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61180716 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133545469277688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26218630 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34951927 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚132851388884992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2424336270848–‚ÿ9w2º - _wandbruntime‚1365  _runtime‚1365u ¼Ú» þ 0loss/FRt2/assignment_loss‚1.4498457079622107 2loss/SRqqt1/assignment_loss‚0.7903127316206408 /loss/FRt1/detection_loss‚0.6134221322663385 1loss/SRqqt2/detection_loss‚0.8224490619139858 /loss/FBt2/detection_loss‚1.2018024033197132 % _timestamp‚1.7844982911325176e+09 _runtime‚1365.046364353 _step‚279 1loss/SRqqt2/assignment_loss‚0.763260500592083 %loss/total_loss‚12.92081608661747 epoch‚4 trainer/global_step‚3449 /loss/FRt1/assignment_loss‚2.965779893483625 /loss/FRt2/detection_loss‚1.4301278465099527 0loss/SRqqt1/detection_loss‚1.293275985836935 /loss/FBt1/detection_loss‚1.3679913322373882 1loss/FBt1/assignment_loss‚0.12255515648702217 1loss/FBt2/assignment_loss‚0.09999333438757674p|? ¼ ƒ /loss/FRt1/assignment_loss‚2.965779893483625 /loss/FRt2/detection_loss‚1.4301278465099527 0loss/SRqqt1/detection_loss‚1.293275985836935 /loss/FBt1/detection_loss‚1.3679913322373882 1loss/FBt1/assignment_loss‚0.12255515648702217 1loss/FBt2/assignment_loss‚0.09999333438757674 0loss/FRt2/assignment_loss‚1.4498457079622107 2loss/SRqqt1/assignment_loss‚0.7903127316206408 /loss/FRt1/detection_loss‚0.6134221322663385 1loss/SRqqt2/detection_loss‚0.8224490619139858 /loss/FBt2/detection_loss‚1.2018024033197132 % _timestamp‚1.7844982911325176e+09 _runtime‚1365.046364353 _step‚279 1loss/SRqqt2/assignment_loss‚0.763260500592083 %loss/total_loss‚12.92081608661747 epoch‚4 trainer/global_step‚3449—y&’Q2½ ‚(:§ ‚‘õÒ±ó¥+& proc.memory.availableMB‚ 196.953125$ disk.nvme1n1.out‚4711.8583984375 disk.dm-0.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 7908.6796875* proc.memory.percent‚17.226672172546387 cpu‚26.165192730014773 proc.cpu.threads‚17) disk./.usagePercent‚75.96640065834083! disk.dm-0.out‚11460.302734375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2822.41796875 network.sent‚ 4.509549e+06! disk.nvme0n1.out‚ 7908.6796875& disk.nvme1c1n1.out‚4711.8583984375 disk.nvme1n1.in‚0 disk.md0.out‚ 41.578125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 network.recv‚439218% disk./.usageGB‚1166.2741661071777 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0! disk.dm-1.out‚1071.5595703125$ memory_percent‚98.79789352416992%¼Iñ›¾ ‚(: ‚‘õÒËý+ gpu.0.gpu‚84.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.353' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àžˆ2¿ - _wandbruntime‚1383  _runtime‚1383@6¼,ÉÀ à % _timestamp‚1.7844983096886976e+09 _runtime‚1383.601381503 _step‚280 ' lr-AdamW/pg1‚0.00010057562776352949 ' lr-AdamW/pg2‚0.00010057562776352949 trainer/global_step‚3499%áÎÁ È ' lr-AdamW/pg1‚0.00010057562776352949 ' lr-AdamW/pg2‚0.00010057562776352949 trainer/global_step‚3499 % _timestamp‚1.7844983096886976e+09 _runtime‚1383.601381503 _step‚280˜³’‹2 - _wandbruntime‚1383  _runtime‚1383ÊøbÒ'à ÌO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134682340879360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62026285 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35448924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35438781 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2529485196800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7844983096892612e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2449912610132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24362001 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529308977664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132935441511760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26577359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128651278783488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8968256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137139936782336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2457595902976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62016140 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131180763980288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26577361 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35448924 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134682357918720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131180562857472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚281 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128651253879808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2449998758606 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134682357918720 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137139936782336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2457683924992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15393507 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132935458551120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137140041843712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24361760 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137140041843712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135385457309726 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135385354121892 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26577359 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134682340879360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2457683924992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35438781É*shvÄ ÑO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15393507 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132935458551120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137140041843712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24361760 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137140041843712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135385457309726 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135385354121892 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26577359 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134682340879360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2457683924992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35438781 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134682340879360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62026285 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35448924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35438781 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2529485196800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7844983096892612e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2449912610132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24362001 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529308977664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132935441511760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26577359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128651278783488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8968256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137139936782336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2457595902976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes‘"Ô_a.all.peak‚ 6159996928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62016140 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131180763980288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26577361 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35448924 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134682357918720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131180562857472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚281 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128651253879808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2449998758606 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134682357918720 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137139936782336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2457683924992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238™HSe‘2Å - _wandbruntime‚1383  _runtime‚1383í »Ü&Æ ÖM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2450674710162 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62043518 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35448679 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134719724285952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135422902141096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2458362041856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134719724285952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚1383.857165215 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62033359 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35448679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15397797 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8970670 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131216953109504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2530192867328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2530004072448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2458274012672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135423005329042 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134719707246592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137178086327808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24368707 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137178086327808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24368464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128686973940736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35458836 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131217166808064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15397794 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132972330618880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3499 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2458362041856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8970910 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2458274012672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128686949037056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62043518 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62033359 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137177981259264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 % _timestamp‚1.7844983099460351e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132972313579520 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816Š;F°á&Ç ÛM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2450674710162 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62043518 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35448679 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134719724285952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135422902141096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2458362041856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134719724285952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚1383.857165215 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62033359 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35448679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15397797 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8970670 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131216953109504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2530192867328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2530004072448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2458274012672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135423005329042 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134719707246592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137178086327808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24368707 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137178086327808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24368464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128686973940736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35458836 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131217166808064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15397794 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132972330618880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3499 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2458362041856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8970910 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2458274012672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128686949037056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62043518 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62033359 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137177981259264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 % _timestamp‚1.7844983099460351e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132972313579520 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330šʧ)ï2È - _wandbruntime‚1383  _runtime‚1383Î`v¬É þ 1loss/FBt1/assignment_loss‚0.30971635615946397 /loss/FRt1/detection_loss‚0.6095270438019909 0loss/FRt2/assignment_loss‚1.1193454414606094 1loss/FBt2/assignment_loss‚0.18838180268323867 trainer/global_step‚3499 _runtime‚1383.858034047 %loss/total_loss‚12.99565525520171 /loss/FRt1/assignment_loss‚2.946345549657804 0loss/SRqqt1/detection_loss‚1.480207732381453 1loss/SRqqt2/detection_loss‚0.6822181489612072 /loss/FBt1/detection_loss‚1.1279562147660143 % _timestamp‚1.7844983099473858e+09 /loss/FRt2/detection_loss‚1.5211376803184764 .loss/FBt2/detection_loss‚1.567485805459008 epoch‚4 _step‚283 2loss/SRqqt1/assignment_loss‚0.7283706717277155 2loss/SRqqt2/assignment_loss‚0.7149628078247284J¡‚b Ê ƒ 2loss/SRqqt1/assignment_loss‚0.7283706717277155 2loss/SRqqt2/assignment_loss‚0.7149628078247284 1loss/FBt1/assignment_loss‚0.30971635615946397 /loss/FRt1/detection_loss‚0.6095270438019909 0loss/FRt2/assignment_loss‚1.1193454414606094 1loss/FBt2/assignment_loss‚0.18838180268323867 trainer/global_step‚3499 _runtime‚1383.858034047 %loss/total_loss‚12.99565525520171 /loss/FRt1/assignment_loss‚2.946345549657804 0loss/SRqqt1/detection_loss‚1.480207732381453 1loss/SRqqt2/detection_loss‚0.6822181489612072 /loss/FBt1/detection_loss‚1.1279562147660143 % _timestamp‚1.7844983099473858e+09 /loss/FRt2/detection_loss‚1.5211376803184764 .loss/FBt2/detection_loss‚1.567485805459008 epoch‚4 _step‚283›Ðþtþ6Ë ‚(:« ‘‘õÒäÅø+ network.sent‚ 4.575374e+06% disk./.usageGB‚1166.2741737365723! disk.dm-1.out‚1071.7080078125$ disk.nvme0n1.out‚7909.7138671875 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 42.07421875 cpu‚26.16340776654241 network.recv‚443750) disk./.usagePercent‚75.96640115528886 disk.dm-0.in‚0& disk.nvme0c0n1.out‚7909.7138671875 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 196.921875# proc.memory.rssMB‚ 2822.41796875! disk.dm-0.out‚11460.869140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme1c1n1.out‚4712.591796875# disk.nvme1n1.out‚4712.591796875# memory_percent‚98.7980842590332* proc.memory.percent‚17.226672172546387 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0Mïï›Ì ‚(: ‘‘õÒ, gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚191.911' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.97033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ž_¯2Í - _wandbruntime‚1402  _runtime‚1402ô?¢õÉÎ Ã _step‚284 ' lr-AdamW/pg1‚0.00010053159742154595 ' lr-AdamW/pg2‚0.00010053159742154595 trainer/global_step‚3549 % _timestamp‚1.7844983285239224e+09 _runtime‚1402.435617965¡nÎÏ È ' lr-AdamW/pg1‚0.00010053159742154595 ' lr-AdamW/pg2‚0.00010053159742154595 trainer/global_step‚3549 % _timestamp‚1.7844983285239224e+09 _runtime‚1402.435617965 _step‚284œ¶¬I2Ð - _wandbruntime‚1402  _runtime‚1402Ç«ÝCÑ ÌO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139042297947136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139042192885760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26943409 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134779044899760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35945676 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2564412568064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137262786985300 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26943409 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2491533644800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24699719 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133000449213952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136550676280320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26943411 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9091712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15608004 aMDeviceStatsMonitor.on_train_batch_start/inactiveÁ$è_split_bytes.large_pool.freed‚130436011742208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2564232154624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2483828234014 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 trainer/global_step‚3549 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35945676 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139042192885760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62878942 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2483742085540 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62889087 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24699469 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136550659240960 _runtime‚1402.435678005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15608007 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137262890173134 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134779061939120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7844983285246894e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139042297947136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2491621666816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133000243896832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139042297947136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139042192885760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26943409 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134779044899760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35945676 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2564412568064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137262786985300 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26943409 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2491533644800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24699719 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133000449213952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136550676280320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26943411 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9091712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15608004 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130436011742208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2564232154624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2483828234014 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 trainer/global_step‚3549 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35945676 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139042192885760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62878942 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2483742085540 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62889087 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24699469 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136550659240960 _runtime‚1402.435678005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15608007 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137262890173134 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134779061939120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7844983285246894e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104¶Á52Ó - _wandbruntime‚1402  _runtime‚1402¬éÀ Ú&Ô ÔM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26950732 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62906320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134815934006880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844983287879238e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26950730 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2492211754496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134815916967520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2484418036984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35955588 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62896161 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24706153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2492299783680 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9094099 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2484504185570 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 1402.6990634 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚3549 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133036846151680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62896161 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133036628258816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137300438192450 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35955588 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136588025608192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136588025608192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2492211754496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139080237362688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26950732 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35945431 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2564921359360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24706396 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139080342431232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26950730 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9093859 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130471706899456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133036628258816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137300438192450 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35955588 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136588025608192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136588025608192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2492211754496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139080237362688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26950732 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35945431 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2564921359360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24706396 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139080342431232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26950730 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9093859 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130471706899456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62906320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134815934006880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844983287879238e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26950730 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2492211754496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134815916967520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2484418036984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35955588 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62896161 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24706153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751ž*XžÜ2Ö - _wandbruntime‚1402  _runtime‚1402mK€× € 0loss/FRt2/assignment_loss‚1.4955483572386996 2loss/SRqqt1/assignment_loss‚0.6583506250653496 1loss/FBt2/assignment_loss‚0.10197170662270309 /loss/FRt2/detection_loss‚1.5733395381677118 % _timestamp‚1.7844983287889373e+09 /loss/FRt1/detection_loss‚0.5998709432806683 _step‚287 0loss/FRt1/assignment_loss‚2.8813835426517156 1loss/FBt1/assignment_loss‚0.24854784382018952 1loss/SRqqt2/detection_loss‚0.7073142946872878 /loss/FBt1/detection_loss‚1.0328458525567876 /loss/FBt2/detection_loss‚0.9370770048999527 %loss/total_loss‚12.33345772966415 trainer/global_step‚3549 1loss/SRqqt2/assignment_loss‚0.737269011038482 1loss/SRqqt1/detection_loss‚1.3599390096346038 epoch‚4 _runtime‚1402.699778441ç¹I÷ Ø … /loss/FRt1/detection_loss‚0.5998709432806683 _step‚287 0loss/FRt1/assignment_loss‚2.8813835426517156 1loss/FBt1/assignment_loss‚0.24854784382018952 1loss/SRqqt2/detection_loss‚0.7073142946872878 /loss/FBt1/detection_loss‚1.0328458525567876 /loss/FBt2/detection_loss‚0.9370770048999527 %loss/total_loss‚12.33345772966415 trainer/global_step‚3549 1loss/SRqqt2/assignment_loss‚0.737269011038482 1loss/SRqqt1/detection_loss‚1.3599390096346038 epoch‚4 _runtime‚1402.699778441 0loss/FRt2/assignment_loss‚1.4955483572386996 2loss/SRqqt1/assignment_loss‚0.6583506250653496 1loss/FBt2/assignment_loss‚0.10197170662270309 /loss/FRt2/detection_loss‚1.5733395381677118 % _timestamp‚1.7844983287889373e+09Ÿj]ò!6Ù ‚(:«  ‘õÒîÎ×+ disk.dm-1.in‚0$ memory_percent‚99.19471740722656# proc.memory.rssMB‚ 2822.41796875 network.sent‚ 4.623835e+06 disk.rbd0.out‚0% disk.nvme1c1n1.out‚4713.232421875# disk.nvme1n1.out‚4713.232421875 disk.md0.out‚ 42.43359375 disk.dm-0.in‚0! disk.dm-0.out‚11463.294921875! disk.dm-1.out‚1071.8173828125 network.recv‚448164 disk.nvme0c0n1.in‚0 disk.md0.in‚0$ proc.memory.availableMB‚131.9375 cpu‚26.160992716060452 proc.cpu.threads‚17) disk./.usagePercent‚75.96642078473579% disk./.usageGB‚1166.2744750976562 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0* proc.memory.percent‚17.226672172546387 disk.rbd0.in‚0& disk.nvme0c0n1.out‚7912.3974609375 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚7912.3974609375ZùyÚ ‚(:„  ‘õҸʸ, gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.224' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.408 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ØàP2Û - _wandbruntime‚1420  _runtime‚14205WäÈÜ Â ' lr-AdamW/pg1‚0.00010048677178785185 ' lr-AdamW/pg2‚0.00010048677178785185 trainer/global_step‚3599 $ _timestamp‚1.784498347027405e+09 _runtime‚1420.938338992 _step‚288z°ÎÍÝ Ç ' lr-AdamW/pg2‚0.00010048677178785185 trainer/global_step‚3599 $ _timestamp‚1.784498347027405e+09 _runtime‚1420.938338992 _step‚288 ' lr-AdamW/pg1‚0.00010048677178785185 kAK2Þ - _wandbruntime‚1420  _runtime‚1420z,DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27309461 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2599371743744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚3599 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36432285 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9214816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138418994641920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132220794508288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2525471386624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134820166252032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140944554050560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784498347028242e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27309461 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138418977602560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36442428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9214573 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2599191330304 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140944448989184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15822507 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132220769604608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2525559408640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136622648287760@–8cÖ'à ÐO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138418977602560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36442428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9214573 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2599191330304 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140944448989184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15822507 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132220769604608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2525559408640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136622648287760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2525471386624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27309461 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2599371743744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚3599 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36432285 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9214816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138418994641920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132220794508288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2525471386624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134820166252032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140944554050560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784498347028242e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27309461 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808¡˜„T2á - _wandbruntime‚1421  _runtime‚1421û}s¶Pâ ÖM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2526237525504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63769122 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132256464761856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2599890664448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27316782 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25043831 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36442183 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63769122 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138456361009152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139177871055858 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚1421.195609071 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36452340 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136659520355520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7844983472845194e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25044083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27316782 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134856562833408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27316780 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27316780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136659537394880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36452340 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15826797 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138456343969792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138456361009152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2526149496320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139177767867912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2518333660978 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9217286 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _step‚290 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140982493466112 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36442183 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚3599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140982598534656 G:DeviceStatsMonitor.,z Œ on_train_batch_end/allocation.all.freed‚63758963 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2600073167872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2526237525504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138456343969792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27316782 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134856562833408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27316780 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27316780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136659537394880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36452340 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15826797 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138456343969792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138456361009152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2526149496320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139177767867912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2518333660978 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9217286 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _step‚290 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140982493466112 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36442183 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚3599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140982598534656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63758963 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2600073167872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2526237525504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138456343969792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63769122 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132256464761856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2599890664448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27316782 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25043831 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36442183 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63769122 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138456361009152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139177871055858 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104¢¼‹2ä - _wandbruntime‚1421  _runtime‚1421‡‘€å  &loss/total_loss‚12.430882216845788 2loss/SRqqt1/assignment_loss‚0.8278674829751254 2loss/SRqqt2/assignment_loss‚0.7042156336093144 2loss/FBt2/assignment_loss‚0.033072868742427396 0loss/FRt1/assignment_loss‚2.8289207835089076 /loss/FRt2/detection_loss‚1.4537853133673828 epoch‚4 trainer/global_step‚3599 % _timestamp‚1.7844983472854884e+09 _runtime‚1421.196128742 0loss/FRt2/assignment_loss‚1.3332859726339683 /loss/FRt1/detection_loss‚0.6625031759472304 1loss/SRqqt2/detection_loss‚0.7303992338682848 -loss/FBt1/detection_loss‚1.29570668409627 /loss/FBt2/detection_loss‚0.9657894629934163 _step‚291 1loss/FBt1/assignment_loss‚0.27128654699754406 1loss/SRqqt1/detection_loss‚1.3240490581059168¾] æ † /loss/FRt1/detection_loss‚0.6625031759472304 1loss/SRqqt2/detection_loss‚0.7303992338682848 -loss/FBt1/detection_loss‚1.29570668409627 /loss/FBt2/detection_loss‚0.9657894629934163 _step‚291 1loss/FBt1/assignment_loss‚0.27128654699754406 1loss/SRqqt1/detection_loss‚1.3240490581059168 &loss/total_loss‚12.430882216845788 2loss/SRqqt1/assignment_loss‚0.8278674829751254 2loss/SRqqt2/assignment_loss‚0.7042156336093144 2loss/FBt2/assignment_loss‚0.033072868742427396 0loss/FRt1/assignment_loss‚2.8289207835089076 /loss/FRt2/detection_loss‚1.4537853133673828 epoch‚4 trainer/global_step‚3599 % _timestamp‚1.7844983472854884e+09 _runtime‚1421.196128742 0loss/FRt2/assignment_loss‚1.3332859726339683£Ï!rÜ8ç ‚(:­ ¯‘õÒÛÄÂ+ disk.nvme1n1.in‚0% disk.nvme1n1.out‚4713.76025390625! disk.dm-0.out‚11463.880859375 disk.md0.out‚ 42.734375$ memory_percent‚99.19390678405762' proc.memory.availableMB‚ 132.0703125* proc.memory.percent‚17.226672172546387 cpu‚26.15897210362115 proc.cpu.threads‚17' disk.nvme1c1n1.out‚4713.76025390625 disk.md0.in‚0# proc.memory.rssMB‚ 2822.41796875 network.sent‚ 4.686358e+06 network.recv‚452050( disk./.usagePercent‚75.9664145728855% disk./.usageGB‚1166.2743797302246 disk.dm-1.in‚0" disk.dm-1.out‚1071.95556640625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚7913.255859375 disk.nvme0n1.in‚0# disk.nvme0n1.out‚7913.255859375ÉK‡¼™è ‚(:Ž ¯‘õÒ׫–, gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚194.104' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.70133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù1à2é - _wandbruntime‚1439  _runtime‚1439)ܨÉê à trainer/global_step‚3649 % _timestamp‚1.7844983658859453e+09 _runtime‚1439.796864089 _step‚292 ' lr-AdamW/pg1‚0.00010044115158042074 ' lr-AdamW/pg2‚0.00010044115158042074UëõÎë È _step‚292 ' lr-AdamW/pg1‚0.00010044115158042074 ' lr-AdamW/pg2‚0.00010044115158042074 trainer/global_step‚3649 % _timestamp‚1.7844983658859453e+09 _runtime‚1439.796864089¤Eœš?2ì - _wandbruntime‚1439  _runtime‚1439œIóìÑ'í ËO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36929037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142846810153984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142846810153984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16037007 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27675509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25374779 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16037004 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134005527467008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36939180 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2559409128448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36939180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138466251675760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚140287313003520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚140287295964160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2551401036356 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27675511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142846705092608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2559497150464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134005552370688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2634158856192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7844983658867996e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9337775 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27675511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141017755899950 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64604546 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25375031 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚3649 _runtime‚ 1439.79760714 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27675509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136639686323200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2634351852544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140287295964160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141017652712116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142846705092608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136639904223232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138466268715120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64614691 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚293 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2559497150464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9338024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚140287313003520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142846705092608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2559497150464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134005552370688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2634158856192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7844983658867996e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9337775 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27675511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141017755899950 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64604546 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25375031 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚3649 _runtime‚ 1439.79760714 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27675509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136639686323200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2634351852544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140287295964160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141017652712116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142846705092608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136639904223232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚ ¤1€0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138466268715120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64614691 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚293 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2559497150464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9338024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚140287313003520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 Bï ‚(:³ ¾‘õÒŸö+* proc.memory.percent‚17.226672172546387 cpu‚26.157487780621395 proc.cpu.threads‚17 disk.md0.in‚0# disk.dm-0.out‚11465.77294921875 network.sent‚ 4.760563e+06) disk./.usagePercent‚75.96643246301434 disk.md0.out‚43.125" disk.dm-1.out‚1072.04541015625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4714.64892578125% disk.nvme1n1.out‚4714.64892578125$ memory_percent‚99.19545650482178% disk./.usageGB‚1166.2746543884277 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7915.19677734375 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2822.41796875 network.recv‚456559 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚7915.19677734375( proc.memory.availableMB‚ 131.81640625Ò¿šð ‚(: ¾‘õÒǦ£, gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚198.694' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.23133333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uñ¯~2ñ - _wandbruntime‚1440  _runtime‚1440ö„w¥Û&ò ÕM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36949092 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2560087238144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27682832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134041222624256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136676287857152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27682830 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9340203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚294 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64631924 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141055303919266 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27682832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141055200731320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844983661432335e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25381497 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36938935 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136676082547200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142884854638080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚ 1440.05443835 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64631924 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142884854638080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140324679370752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2560175267328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25381748 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2552076987800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140324662331392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9340451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27682830 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138503123743520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 Iòžà&ó ÚM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2552076987800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140324662331392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9340451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27682830 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138503123743520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36949092 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2560087238144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27682832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134041222624256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136676287857152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27682830 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9340203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚294 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64631924 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141055303919266 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27682832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141055200731320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844983661432335e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25381497 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36938935 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136676082547200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142884854638080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚ 1440.05443835 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64631924 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142884854638080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140324679370752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2560175267328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25381748 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374¦éh¡2ô - _wandbruntime‚1440  _runtime‚1440Néõ ü /loss/FRt2/detection_loss‚1.5042749901982884 % _timestamp‚1.7844983661443653e+09 /loss/FRt2/assignment_loss‚1.384508008937759 1loss/FBt2/assignment_loss‚0.00880402529645942 /loss/FRt1/detection_loss‚0.5665531022923991 0loss/SRqqt1/detection_loss‚1.376411027384789 1loss/SRqqt2/detection_loss‚0.7987902197960587 &loss/total_loss‚18.125738471182586 trainer/global_step‚3649 1loss/FBt1/assignment_loss‚0.25763517581835915 epoch‚4 _step‚295 2loss/SRqqt1/assignment_loss‚0.8892807713310633 2loss/SRqqt2/assignment_loss‚0.7872367212904662 .loss/FBt1/detection_loss‚3.096722641699792 .loss/FBt2/detection_loss‚4.385717433340769 _runtime‚1440.054974721 .loss/FRt1/assignment_loss‚3.06980435379638öî—3ö  /loss/FRt1/detection_loss‚0.5665531022923991 0loss/SRqqt1/detection_loss‚1.376411027384789 1loss/SRqqt2/detection_loss‚0.7987902197960587 &loss/total_loss‚18.125738471182586 trainer/global_step‚3649 1loss/FBt1/assignment_loss‚0.25763517581835915 epoch‚4 _step‚295 2loss/SRqqt1/assignment_loss‚0.8892807713310633 2loss/SRqqt2/assignment_loss‚0.7872367212904662 .loss/FBt1/detection_loss‚3.096722641699792 .loss/FBt2/detection_loss‚4.385717433340769 _runtime‚1440.054974721 .loss/FRt1/assignment_loss‚3.06980435379638 /loss/FRt2/detection_loss‚1.5042749901982884 % _timestamp‚1.7844983661443653e+09 /loss/FRt2/assignment_loss‚1.384508008937759 1loss/FBt2/assignment_loss‚0.00880402529645942§é /÷ ‚(:¤ Í‘õÒ½Àæ+ cpu‚26.15571297319316" disk.dm-1.out‚1072.22412109375 disk.rbd0.out‚0 disk.nvme0n1.in‚0 memory_percent‚ 99.1943359375 proc.cpu.threads‚17( disk./.usagePercent‚75.9664337053844$ disk./.usageGB‚1166.274673461914 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 proc.memory.availableMB‚132 network.sent‚ 4.812076e+06 disk.dm-1.in‚0' disk.nvme0c0n1.out‚7916.09619140625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4715.36767578125% disk.nvme1n1.out‚4715.36767578125 disk.md0.out‚43.5625 network.recv‚461144# disk.dm-0.out‚11466.26513671875% disk.nvme0n1.out‚7916.09619140625# proc.memory.rssMB‚ 2822.41796875* proc.memory.percent‚17.226672172546387³Ž2]›ø ‚(: Í‘õÒä…”, gpu.0.gpu‚86.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚114.689' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.22966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04p‰Ê2ù - _wandbruntime‚1458  _runtime‚1458øßÉú à ' lr-AdamW/pg2‚0.00010039473752995282 trainer/global_step‚3699 % _timestamp‚1.7844983847021253e+09 _runtime‚1458.613054594 _step‚296 ' lr-AdamW/pg1‚0.00010039473752995282~§pÎû È % _timestamp‚1.7844983847021253e+09 _runtime‚1458.613054594 _step‚296 ' lr-AdamW/pg1‚0.00010039473752995282 ' lr-AdamW/pg2‚0.00010039473752995282 trainer/global_step‚3699¨¨é"2ü - _wandbruntime‚1458  _runtime‚1458“ï5ÝKý ÌO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142155614325760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9460864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2669281942016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9460602 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140309855063760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2593434892288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2593434892288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2669105722880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_Ž´‡batch_start/requested_bytes.all.freed‚142895085575524 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2585316660238 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1458.613764206 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2593346870272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16251507 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2585230511764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65467348 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138459391052288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚140309872103120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2593346870272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135790310233088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25712106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135790285329408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3699 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144748961196032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37425789 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28041561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142155631365120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25712371 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28041561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142895188763358 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144749066257408·ö6à×'þ ÑO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144748961196032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37425789 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28041561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142155631365120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25712371 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28041561 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142895188763358 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144749066257408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142155614325760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9460864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2669281942016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9460602 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140309855063760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2593434892288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2593434892288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2669105722880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142895085575524 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2585316660238 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1458.613764206 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2593346870272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16251507 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2585230511764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65467348 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138459391052288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚140309872103120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2593346870272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135790310233088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25712106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135790285329408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3699©Üž»_2ÿ - _wandbruntime‚1458  _runtime‚1458 ÖŸAÜ&€ ÖM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142932633594728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142192980692992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28048880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37435687 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1458.871414759 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28048882 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25718806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2585906463208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142192997732352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37445844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25719053 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65494726 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2594113009152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚298 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2594113009152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142932736782674 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65494726 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37435687 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138495790137344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 % _timestamp‚1.7844983849602473e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135826005390336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚3699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9463256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142192980692992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9463012 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16255797 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37445844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28048882 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135825980486656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2669809650688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚298 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2594113009152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142932736782674 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65494726 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37435687 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138495790137344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 % _timestamp‚1.7844983849602473e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135826005390336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚3699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9463256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142192980692992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9463012 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16255797 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37445844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28048882 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135825980486656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2669809650688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142932633594728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142192980692992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28048880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37435687 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1458.871414759 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28048882 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25718806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ªsŸ›a2‚ - _wandbruntime‚1458  _runtime‚1458ž_ékƒ ý 1loss/FBt2/assignment_loss‚0.05437111463732928 1loss/SRqqt2/detection_loss‚0.7658093792528471 &loss/total_loss‚13.569882492801773 /loss/FRt1/assignment_loss‚2.878762097422141 0loss/FRt2/assignment_loss‚1.2366806213706183 0loss/FBt1/assignment_loss‚0.1437071878473424 /loss/FRt1/detection_loss‚0.6118568905033976 _step‚299 2loss/SRqqt1/assignment_loss‚0.8120967649173603 1loss/SRqqt1/detection_loss‚1.5405308823170725 /loss/FBt2/detection_loss‚1.2272021887555968 % _timestamp‚1.7844983849617538e+09 -loss/FRt2/detection_loss‚1.52214111258427 .loss/FBt1/detection_loss‚2.083039429439287 epoch‚4 trainer/global_step‚3699 _runtime‚1458.872446611 2loss/SRqqt2/assignment_loss‚0.6936848237545097†ÜTQ„ ‚ 1loss/SRqqt2/detection_loss‚0.7658093792528471 &loss/total_loss‚13.569882492801773 /loss/FRt1/assignment_loss‚2.878762097422141 0loss/FRt2/assignment_loss‚1.2366806213706183 0loss/FBt1/assignment_loss‚0.1437071878473424 /loss/FRt1/detection_loss‚0.6118568905033976 _step‚299 2loss/SRqqt1/assignment_loss‚0.8120967649173603 1loss/SRqqt1/detection_loss‚1.5405308823170725 /loss/FBt2/detection_loss‚1.2272021887555968 % _timestamp‚1.7844983849617538e+09 -loss/FRt2/detection_loss‚1.52214111258427 .loss/FBt1/detection_loss‚2.083039429439287 epoch‚4 trainer/global_step‚3699 _runtime‚1458.872446611 2loss/SRqqt2/assignment_loss‚0.6936848237545097 1loss/FBt2/assignment_loss‚0.05437111463732928«R-Ø*5… ‚(:ª Ü‘õÒ»ƒù+# proc.memory.rssMB‚ 2822.41796875* proc.memory.percent‚17.226672172546387 cpu‚26.15413276244898" disk.dm-1.out‚1072.33740234375 disk.rbd0.out‚0 network.sent‚ 4.864004e+06 disk.nvme1n1.in‚0# disk.nvme1n1.out‚4716.197265625 disk.md0.in‚0% disk.nvme1c1n1.out‚4716.197265625$ memory_percent‚99.19471740722656 network.recv‚465330 disk.dm-0.in‚0# disk.dm-0.out‚11470.41162109375 disk.rbd0.in‚0# disk.nvme0n1.out‚7920.037109375 proc.cpu.threads‚17) disk./.usagePercent‚75.95095899240569% disk./.usageGB‚1166.0370979309082 disk.md0.out‚ 43.8046875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚7920.037109375$ proc.memory.availableMB‚131.9375 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0ÅY¸{›† ‚(: Ü‘õÒÈÜ , gpu.0.gpu‚83.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚201.529' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.17633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 â„2‡ - _wandbruntime‚1477  _runtime‚1477M&ËÜɈ à ' lr-AdamW/pg2‚0.00010034753037986331 trainer/global_step‚3749 % _timestamp‚1.7844984035298681e+09 _runtime‚1477.440805034 _step‚300 ' lr-AdamW/pg1‚0.00010034753037986331×6v‰Î‰ È _runtime‚1477.440805034 _step‚300 ' lr-AdamW/pg1‚0.00010034753037986331 ' lr-AdamW/pg2‚0.00010034753037986331 trainer/global_step‚3749 % _timestamp‚1.7844984035298681e+09¬"ü®ú2Š - _wandbruntime‚1477  _runtime‚1477â8­Ò'‹ ÌO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16466007 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146651217299456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140279088824320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2627372634112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37932684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37922541 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26049669 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137575068095488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137575043191808 trainer/global_step‚3749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144023932687360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66340295 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146651322360832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2627372634112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142153458451760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66330150 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28407609 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144023949726720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9583898 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142153475491120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144023932687360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7844984035307968e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37922541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2619059987172 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚1477.441719905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16466004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144772621626766 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2627284612096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146651217299456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144772518438932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627284612096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28407611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146651217299456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144772518438932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627284612096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28407611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16466007 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146651217299456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140279088824320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2627372634112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37932684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37922541 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26049669 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137575068095488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137575043191808 trainer/global_step‚3749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144023932687360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66340295 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146651322360832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2627372634112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142153458451760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66330150 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28407609 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144023949726720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9583898 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142153475491120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144023932687360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7844984035307968e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37922541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2619059987172 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚1477.441719905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16466004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144772621626766­Ù=Äà2 - _wandbruntime‚1477  _runtime‚1477Û6ýKŽ ÖM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144061316093952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28414932 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9586009 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2704749844992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142190330519520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146689261776384 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66357528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144810066458136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137610763252736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37942596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66357528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚302 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26056303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137610738349056 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28414930 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37942596 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated¶½,'‘‚140315689309696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2619735938616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26056535 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28414930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66347369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16470297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16470294 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144061299054592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146689366844928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37932439 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28414932 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140315488194048 % _timestamp‚1.7844984037904053e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2704926056960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627962721792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146689366844928 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚3749 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144810169646082 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816a;Þá& ÛM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26056303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137610738349056 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28414930 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37942596 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140315689309696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2619735938616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26056535 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28414930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66347369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16470297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16470294 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144061299054592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146689366844928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37932439 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28414932 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140315488194048 % _timestamp‚1.7844984037904053e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2704926056960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627962721792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146689366844928 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚3749 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144810169646082 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144061316093952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28414932 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9586009 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2704749844992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142190330519520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146689261776384 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66357528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144810066458136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137610763252736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37942596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66357528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚302®&72 - _wandbruntime‚1477  _runtime‚1477LXŸ‘ þ trainer/global_step‚3749 $ _timestamp‚1.784498403791354e+09 1loss/SRqqt2/assignment_loss‚0.718213425458978 /loss/FRt1/detection_loss‚0.6291023439357335 &loss/total_loss‚13.798882260531482 0loss/FRt1/assignment_loss‚2.8971521142085037 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7827903665152603 1loss/FBt1/assignment_loss‚0.29737460600119026 /loss/FRt2/detection_loss‚1.4248216691317768 1loss/SRqqt1/detection_loss‚1.3633169905635818 1loss/SRqqt2/detection_loss‚0.7211562384136616 /loss/FBt1/detection_loss‚2.7580708915797563 _runtime‚1477.701976467 _step‚303 /loss/FRt2/assignment_loss‚1.165752308709281 2loss/FBt2/assignment_loss‚0.011572400833849732 -loss/FBt2/detection_loss‚1.02955890517991 ÑaH ’ ƒ 1loss/SRqqt1/detection_loss‚1.3633169905635818 1loss/SRqqt2/detection_loss‚0.7211562384136616 /loss/FBt1/detection_loss‚2.7580708915797563 _runtime‚1477.701976467 _step‚303 /loss/FRt2/assignment_loss‚1.165752308709281 2loss/FBt2/assignment_loss‚0.011572400833849732 -loss/FBt2/detection_loss‚1.02955890517991 trainer/global_step‚3749 $ _timestamp‚1.784498403791354e+09 1loss/SRqqt2/assignment_loss‚0.718213425458978 /loss/FRt1/detection_loss‚0.6291023439357335 &loss/total_loss‚13.798882260531482 0loss/FRt1/assignment_loss‚2.8971521142085037 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7827903665152603 1loss/FBt1/assignment_loss‚0.29737460600119026 /loss/FRt2/detection_loss‚1.4248216691317768¯zÚ£åS“ ‚(:È ë‘õÒœ×Ñ+ cpu‚26.152433888950203 proc.cpu.threads‚17 disk.md0.out‚ 44.3984375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚4717.0830078125 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2822.41796875* proc.memory.percent‚17.226672172546387 network.sent‚ 4.913353e+06 disk.md0.in‚0$ disk.nvme1n1.out‚4717.0830078125 network.recv‚469860 disk.dm-0.in‚ 0.0078125 disk.dm-1.in‚0" disk.dm-1.out‚1072.50537109375 disk.nvme0c0n1.in‚ 0.0078125& disk.nvme0c0n1.out‚8251.9931640625 disk.nvme0n1.in‚ 0.0078125 disk.nvme1c1n1.in‚0) disk./.usagePercent‚75.95045782032395% disk./.usageGB‚1166.0294036865234# disk.dm-0.out‚11801.85302734375$ disk.nvme0n1.out‚8251.9931640625$ memory_percent‚99.19490814208984% proc.memory.availableMB‚ 131.90625j¼.ƒš” ‚(: ë‘õÒÖù¤, gpu.0.gpu‚57.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.579' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.52633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$º°2• - _wandbruntime‚1495  _runtime‚1495Éâ+É– à _runtime‚1495.941962795 _step‚304 ' lr-AdamW/pg1‚0.00010029953088627054 ' lr-AdamW/pg2‚0.00010029953088627054 trainer/global_step‚3799 % _timestamp‚1.7844984220310013e+09A<ÃÇΗ È _runtime‚1495.941962795 _step‚304 ' lr-AdamW/pg1‚0.00010029953088627054 ' lr-AdamW/pg2‚0.00010029953088627054 trainer/global_step‚3799 % _timestamp‚1.7844984220310013e+09°¦N\Î2˜ - _wandbruntime‚1495  _runtime‚1495‘¤Ò'™ ÌO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67192952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148553578464256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142098746818560 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2652889462580 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28773659 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419293 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚305 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16680507 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148553578464256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67203097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139359801054208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2738945764352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26386994 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2661310375936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2652975611054 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2661222353920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2661222353920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28773659 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9706245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139359825957888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145892268088320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148553473402880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28773661 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9706487 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16680504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145892251048960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419293 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚3799 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145892268088320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145892251048960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429436 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429436 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2739128274944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148553473402880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26386749¬úD:š ÑO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429436 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429436 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2739128274944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148553473402880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26386749 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67192952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148553578464256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142098746818560 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2652889462580 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28773659 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419293 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚305 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16680507 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148553578464256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67203097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139359801054208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2738945764352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26386994 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2661310375936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2652975611054 MGDeviceStatsM‚ onitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2661222353920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2661222353920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28773659 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9706245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139359825957888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145892268088320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148553473402880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28773661 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9706487 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16680504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145892251048960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419293 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚3799 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145892268088320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145892251048960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816±¢%e²2› - _wandbruntime‚1496  _runtime‚1496XO·Ü&œ ÖM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2661900463616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2653651562610 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26393501 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38429191 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148591622948352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144033950946880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16684797 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28780980 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38439348 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145929634455552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67220330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2739642280448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67210171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148591517879808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2653565414024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9708707 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2739831075328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145929617416192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2661988492800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2661988492800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67210171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚1496.200461508 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145929617416192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139395521115136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38429191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148591622948352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28780982 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146687602509490 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28780982 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146687499321544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844984222893977e+09 _step‚306 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38429191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148591622948352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28780982 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146687602509490 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28780982 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146687499321544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844984222893977e+09 _step‚306 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26393501 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38429191 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148591622948352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144033950946880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16684797 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28780980 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38439348 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145929634455552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67220330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2739642280448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67210171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148591517879808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2653565414024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9708707 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2739831075328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145929617416192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2661988492800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2661988492800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67210171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712²>¼Îm2ž - _wandbruntime‚1496  _runtime‚1496¤ÞÁTŸ ÿ 0loss/SRqqt1/detection_loss‚1.378499784647657 1loss/SRqqt2/detection_loss‚0.8296368093758729 % _timestamp‚1.7844984222903428e+09 .loss/FRt1/assignment_loss‚2.78375105418657 0loss/FRt2/assignment_loss‚1.3018029480115743 2loss/SRqqt1/assignment_loss‚0.6817329415038379 2loss/SRqqt2/assignment_loss‚0.6789794692735153 /loss/FRt1/detection_loss‚0.5888086710032076 /loss/FBt2/detection_loss‚0.7318805828060037 _runtime‚1496.200943629 _step‚307 1loss/FBt1/assignment_loss‚0.09121132877629488 /loss/FBt1/detection_loss‚1.3252278418781667 &loss/total_loss‚11.870904328974637 epoch‚4 1loss/FBt2/assignment_loss‚0.03914539368926653 /loss/FRt2/detection_loss‚1.4402275038226706 trainer/global_step‚3799&¯   „ .loss/FRt1/assignment_loss‚2.78375105418657 0loss/FRt2/assignment_loss‚1.3018029480115743 2loss/SRqqt1/assignment_loss‚0.6817329415038379 2loss/SRqqt2/assignment_loss‚0.6789794692735153 /loss/FRt1/detection_loss‚0.5888086710032076 /loss/FBt2/detection_loss‚0.7318805828060037 _runtime‚1496.200943629 _step‚307 1loss/FBt1/assignment_loss‚0.09121132877629488 /loss/FBt1/detection_loss‚1.3252278418781667 &loss/total_loss‚11.870904328974637 epoch‚4 1loss/FBt2/assignment_loss‚0.03914539368926653 /loss/FRt2/detection_loss‚1.4402275038226706 trainer/global_step‚3799 0loss/SRqqt1/detection_loss‚1.378499784647657 1loss/SRqqt2/detection_loss‚0.8296368093758729 % _timestamp‚1.7844984222903428e+09³âQ¡ ‚(:Æ ú‘õÒÈ÷+ network.sent‚ 4.989395e+06) disk./.usagePercent‚75.95048813415339 disk.dm-1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4718.00732421875$ memory_percent‚99.19512271881104# proc.memory.rssMB‚ 2822.41796875 network.recv‚474188 disk.nvme0c0n1.in‚ 0.0078125# disk.nvme0c0n1.out‚ 8255.8046875 disk.nvme0n1.in‚ 0.0078125 disk.md0.in‚0! disk.dm-0.out‚11805.619140625" disk.dm-1.out‚1072.55615234375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4718.00732421875( proc.memory.availableMB‚ 131.87109375* proc.memory.percent‚17.226672172546387 cpu‚26.150746945684784% disk./.usageGB‚1166.0298690795898 disk.md0.out‚ 44.82421875 disk.dm-0.in‚ 0.0078125 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.out‚ 8255.8046875 proc.cpu.threads‚17ÜDÈÈ™¢ ‚(:Ž ú‘õÒô¿¦, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.845' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.61500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o-jK£ ‚(:À ‰’õÒ¥ÁÈ+ disk.md0.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚8256.580078125 disk.nvme0n1.in‚ 0.0078125 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 131.87109375 disk.dm-0.in‚ 0.0078125! disk.dm-0.out‚11806.103515625$ memory_percent‚99.19512271881104# proc.memory.rssMB‚ 2822.41796875* proc.memory.percent‚17.226672172546387 proc.cpu.threads‚17$ disk./.usageGB‚1166.029556274414 disk.nvme1n1.in‚0# disk.nvme1n1.out‚4718.529296875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.0078125# disk.nvme0n1.out‚8256.580078125 cpu‚26.16204112169921 network.sent‚ 5.039623e+06 disk.md0.out‚ 45.140625 disk.dm-1.in‚0 disk.dm-1.out‚1072.685546875% disk.nvme1c1n1.out‚4718.529296875 network.recv‚478405) disk./.usagePercent‚75.95046775928442ë™äœ¤ ‚(:‘ ‰’õÒ„±ø+ gpu.0.gpu‚87.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚113.291' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.763666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kU`92¥ - _wandbruntime‚1515  _runtime‚1515}…Ȧ  _runtime‚1515.192335587 _step‚308 ' lr-AdamW/pg1‚0.00010025073981798386 ' lr-AdamW/pg2‚0.00010025073981798386 trainer/global_step‚3849 $ _timestamp‚1.784498441280552e+09%DËMͧ Ç _runtime‚1515.192335587 _step‚308 ' lr-AdamW/pg1‚0.00010025073981798386 ' lr-AdamW/pg2‚0.00010025073981798386 trainer/global_step‚3849 $ _timestamp‚1.784498441280552e+09´é¡,G2¨ - _wandbruntime‚1515  _runtime‚1515†ò$ ù© ËO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38926188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29139709 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29139711 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150455729506304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147760569410560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2686805086462 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38916045 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2695160095744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150455729506304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_Š¥Øbatch_start/active_bytes.all.allocated‚150455834567680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143918702262272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141144558916608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2774118441984 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚3849 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147760569410560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26724331 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9829571 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9829327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68055754 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚1515.192388927 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147760586449920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2695248117760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145840665227760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29139709 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141144583820288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150455834567680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16895004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 $ _timestamp‚1.784498441281197e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29139711 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2695248117760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2686718937988 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38916045 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16895007 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26724578 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482ê`V•Ö'ª ÐO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2695160095744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148527487353582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145840682267120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68065899 _step‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143918490653696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148527384165748 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38926188 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38926188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29139709 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29139711 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150455729506304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147760569410560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2686805086462 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38916045 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2695160095744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150455729506304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150455834567680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143918702262272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141144558916608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2774118441984 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚3849 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147760569410560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26724331 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9829571 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9829327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68055754 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚1515.192388927 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147760586449920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2695248117760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145840665227760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29139709 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141144583820288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150455834567680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16895004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 $ _timestamp‚1.784498441281197e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29139711 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2695248117760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2686718937988 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38916045 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16895007 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26724578 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68083132 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68072973 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26731045 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147797935777792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2687481038018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68083132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143954881647104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774812173824 _runtime‚ 1515.45036574 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150493879051776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141180254073856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148565035372898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2687394889432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150493879051776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141180278977536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9832009 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38925943 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29147032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147797952817152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16899297 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145877554334880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2695838205440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38925943 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147797952817152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147797935777792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2695926234624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29147032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29147030 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38936100 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150493773983232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2695926234624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚3849 % _timestamp‚1.7844984415392075e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68072973 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145877537295520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774627573248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26731306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16899294 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104C‰¸¿"­ ÚM trainer/global_step‚3849 % _timestamp‚1.7844984415392075e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68072973 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145877537295520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774627573248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26731306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16899294 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68083132 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68072973 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26731045 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147797935777792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2687481038018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68083132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143954881647104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774812173824 _runtime‚ 1515.45036574 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150493879051776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_allocf‹zD¾_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141180254073856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148565035372898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2687394889432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150493879051776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141180278977536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9832009 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38925943 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29147032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147797952817152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16899297 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145877554334880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2695838205440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38925943 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147797952817152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147797935777792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2695926234624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29147032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29147030 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38936100 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150493773983232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2695926234624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808¶OÒ2® - _wandbruntime‚1515  _runtime‚1515Å'‹¯ € /loss/FBt2/detection_loss‚0.9596055688488867 trainer/global_step‚3849 _runtime‚1515.450854391 1loss/FBt1/assignment_loss‚0.11924644057391731 0loss/SRqqt1/detection_loss‚1.214230018515959 0loss/FRt2/assignment_loss‚1.2218482418992054 2loss/SRqqt1/assignment_loss‚0.7500779289316307 2loss/SRqqt2/assignment_loss‚0.8054007153456303 /loss/FRt1/detection_loss‚0.6167486485111559 /loss/FRt2/detection_loss‚1.4385535414473394 1loss/SRqqt2/detection_loss‚0.6661110634108041 %loss/total_loss‚11.79541147365423 $ _timestamp‚1.784498441540141e+09 2loss/FBt2/assignment_loss‚0.019550719839500862 epoch‚4 _step‚311 0loss/FRt1/assignment_loss‚2.8134191186845174 /loss/FBt1/detection_loss‚1.1706194676456845º} ° … 0loss/FRt1/assignment_loss‚2.8134191186845174 /loss/FBt1/detection_loss‚1.1706194676456845 /loss/FBt2/detection_loss‚0.9596055688488867 trainer/global_step‚3849 _runtime‚1515.450854391 1loss/FBt1/assignment_loss‚0.11924644057391731 0loss/SRqqt1/detection_loss‚1.214230018515959 0loss/FRt2/assignment_loss‚1.2218482418992054 2loss/SRqqt1/assignment_loss‚0.7500779289316307 2loss/SRqqt2/assignment_loss‚0.8054007153456303 /loss/FRt1/detection_loss‚0.6167486485111559 /loss/FRt2/detection_loss‚1.4385535414473394 1loss/SRqqt2/detection_loss‚0.6661110634108041 %loss/total_loss‚11.79541147365423 $ _timestamp‚1.784498441540141e+09 2loss/FBt2/assignment_loss‚0.019550719839500862 epoch‚4 _step‚311·O¶œ¸O± ‚(:Ä ˜’õÒ˨+ disk.dm-1.out‚1072.798828125 disk.rbd0.out‚0 network.recv‚482132 disk.dm-1.in‚0' disk.nvme0c0n1.out‚8271.99560546875% disk.nvme0n1.out‚8271.99560546875 disk.nvme1n1.in‚0* proc.memory.percent‚17.226672172546387 cpu‚26.172949816242145 proc.cpu.threads‚17 network.sent‚ 5.042646e+06) disk./.usagePercent‚75.95099899672158 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚4720.802734375# disk.nvme1n1.out‚4720.802734375 disk.md0.in‚0 disk.md0.out‚45.375 disk.dm-0.in‚ 0.0078125# disk.dm-0.out‚11823.17138671875 disk.nvme0c0n1.in‚ 0.0078125 disk.nvme0n1.in‚ 0.0078125$ memory_percent‚99.19707775115967( proc.memory.availableMB‚ 131.55078125# proc.memory.rssMB‚ 2822.41796875$ disk./.usageGB‚1166.037712097168RT#›² ‚(: ˜’õÒûè´+ gpu.0.gpu‚90.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚196.015' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.33833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñð2³ - _wandbruntime‚1534  _runtime‚1534稀1Ç´ Á & lr-AdamW/pg1‚0.0001002011579564912 & lr-AdamW/pg2‚0.0001002011579564912 trainer/global_step‚3899 % _timestamp‚1.7844984607225718e+09 _runtime‚1534.633578046 _step‚312Â:Ó̵ Æ % _timestamp‚1.7844984607225718e+09 _runtime‚1534.633578046 _step‚312 & lr-AdamW/pg1‚0.0001002011579564912 & lr-AdamW/pg2‚0.0001002011579564912 trainer/global_step‚3899¸M–Ô/2¶ - _wandbruntime‚1534  _runtime‚15345ªœžÐ'· ÊO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2720634561870 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152358090671104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149628904811520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27062015 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152358090671104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29505761 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152357985609728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _runtime‚ 1534.6356734 _step‚313 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147684285655120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2729097837568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29505759 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150404817029156 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39412797 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142929341682688 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39412797 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68918556 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145738204609536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 % _timestamp‚1.7844984607246459e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68928701 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2729097837568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2808887830528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149628904811520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2729185859584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145738407829504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9952508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2809066146816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17109507 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9952246 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2720548413396 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚3899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152357985609728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142929316779008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29505761 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39422940 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2729185859584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17109504 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68928701 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2729097837568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2808887830528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149628904811520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2729185859584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145738407829504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9952508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2809066146816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17109507 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9952246 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2720548413396 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚3899 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152357985609728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142929316779008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29505761 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39422940 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2729185859584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17109504 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152357985609728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _runtime‚ 1534.6356734 _step‚313 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147684285655120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2729097837568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29505759 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150404817029156 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39412797 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0¹¨:Y”2¹ - _wandbruntime‚1534  _runtime‚1534á­+Ãå º ÕM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29513082 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2809769369088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150442468236306 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149666254139392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39432852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29513082 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152396030086656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145774598801920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68945934 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68945934 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27068526 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17113797 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152396135155200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29513080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9954732 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚314 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147721140683520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 traiJ+Ðöner/global_step‚3899 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39432852 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2729863976448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27068760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39422695 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150442365048360 _runtime‚1534.892305522 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2729775947264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721224364840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2729863976448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142965036839936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142965011936256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9954963 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 580530380853{ªà&» ÚM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2729863976448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27068760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39422695 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150442365048360 _runtime‚1534.892305522 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2729775947264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721224364840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2729863976448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142965036839936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142965011936256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9954963 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29513082 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2809769369088 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150442468236306 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149666254139392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39432852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29513082 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152396030086656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145774598801920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68945934 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68945934 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27068526 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17113797 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152396135155200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29513080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9954732 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚314 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147721140683520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3899 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39432852 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184º4£òK2¼ - _wandbruntime‚1534  _runtime‚1534\¿ ó½ ý /loss/FRt1/assignment_loss‚2.974279196729639 .loss/FBt2/detection_loss‚0.801372729273155 epoch‚4 # _timestamp‚1.78449846098216e+09 2loss/SRqqt1/assignment_loss‚0.6834708640351892 /loss/FRt2/detection_loss‚1.3400400949698128 /loss/FBt1/detection_loss‚1.8600482076481761 _runtime‚1534.892867773 1loss/FBt2/assignment_loss‚0.06688623025065536 1loss/SRqqt2/detection_loss‚0.7184900530535077 _step‚315 0loss/FRt2/assignment_loss‚1.2458224605557113 2loss/SRqqt2/assignment_loss‚0.6488114845139977 1loss/FBt1/assignment_loss‚0.10769202205756301 /loss/FRt1/detection_loss‚0.6160179847238983 1loss/SRqqt1/detection_loss‚1.3263502050723348 %loss/total_loss‚12.38928153288364 trainer/global_step‚3899UÖä¾ ‚ .loss/FBt2/detection_loss‚0.801372729273155 epoch‚4 # _timestamp‚1.78449846098216e+09 2loss/SRqqt1/assignment_loss‚0.6834708640351892 /loss/FRt2/detection_loss‚1.3400400949698128 /loss/FBt1/detection_loss‚1.8600482076481761 _runtime‚1534.892867773 1loss/FBt2/assignment_loss‚0.06688623025065536 1loss/SRqqt2/detection_loss‚0.7184900530535077 _step‚315 0loss/FRt2/assignment_loss‚1.2458224605557113 2loss/SRqqt2/assignment_loss‚0.6488114845139977 1loss/FBt1/assignment_loss‚0.10769202205756301 /loss/FRt1/detection_loss‚0.6160179847238983 1loss/SRqqt1/detection_loss‚1.3263502050723348 %loss/total_loss‚12.38928153288364 trainer/global_step‚3899 /loss/FRt1/assignment_loss‚2.974279196729639»1R2EN¿ ‚(:à §’õÒ®ü+) disk./.usagePercent‚75.95096073172375& disk.nvme1c1n1.out‚4724.7724609375 disk.dm-1.out‚ 1072.9375' disk.nvme0c0n1.out‚8272.72509765625% disk.nvme0n1.out‚8272.72509765625# proc.memory.rssMB‚ 2822.41796875 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 45.6015625 disk.dm-0.in‚ 0.0078125# disk.dm-0.out‚11827.24169921875 disk.dm-1.in‚0$ memory_percent‚99.19729232788086$ disk./.usageGB‚1166.037124633789$ disk.nvme1n1.out‚4724.7724609375 disk.nvme0c0n1.in‚ 0.0078125 disk.nvme0n1.in‚ 0.0078125& proc.memory.availableMB‚ 131.515625 network.sent‚ 5.117976e+06 network.recv‚486680 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.226672172546387 cpu‚26.178642479669538VnFšÀ ‚(: §’õÒúâë+ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚200.757' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.91900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìè2Á - _wandbruntime‚1554  _runtime‚1554 ¿_É Ã _step‚316 ' lr-AdamW/pg1‚0.00010015078609594678 ' lr-AdamW/pg2‚0.00010015078609594678 trainer/global_step‚3949 % _timestamp‚1.7844984802534192e+09 _runtime‚1554.167445282÷÷|¸Îà È % _timestamp‚1.7844984802534192e+09 _runtime‚1554.167445282 _step‚316 ' lr-AdamW/pg1‚0.00010015078609594678 ' lr-AdamW/pg2‚0.00010015078609594678 trainer/global_step‚3949¼›u¯72Ä - _wandbruntime‚1554  _runtime‚1554|5µuÔ'Å ÎO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69781358 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚317 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149527889043120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152282249892564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154260346774528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2763123601408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69791503 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10076605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29871811 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2754464037278 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154260241713152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144714074641408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2763123601408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚1554.168036313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147557891848192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154260346774528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27400609 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2763035579392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151497223173120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151497206133760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27400859 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2754377888804 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7844984802572422e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151497223173120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17324004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39919692 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39909549 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2843995523072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10076852 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152282353080398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29871811 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2763035579392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29871809 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144714099545088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154260241713152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2843817206784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482;­ƒ”Û Æ ÓO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚317 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149527889043120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152282249892564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154260346774528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2763123601408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69791503 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split8‹Æ-þ.small_pool.freed‚10076605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29871811 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2754464037278 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154260241713152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144714074641408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2763123601408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚1554.168036313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147557891848192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154260346774528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27400609 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2763035579392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151497223173120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151497206133760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27400859 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2754377888804 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7844984802572422e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151497223173120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17324004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39919692 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39909549 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2843995523072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10076852 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152282353080398 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29871811 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2763035579392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29871809 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144714099545088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154260241713152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2843817206784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69781358 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374½ï6}2Ç - _wandbruntime‚1554  _runtime‚1554g² øÞ&È ØM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29879132 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151534572500992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149564761110880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2755139988834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154298391258624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2763713689088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29879130 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147594283279872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _runtime‚1554.427075846 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69808736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27407340 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149564744071520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2763801718272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚3949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39929604 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2755053840248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154298286190080 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151534589540352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29879130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29879132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152319797911768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17328297 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚154298391258624 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39919447 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27407593 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39919447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚147594492784128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2844513481216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151534589540352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151534572500992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69798577 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144749794702336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69808736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152ðÂ+ã&É ÝM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39919447 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27407593 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39919447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚147594492784128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2844513481216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151534589540352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151534572500992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69798577 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144749794702336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69808736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29879132 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151534572500992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149564761110880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2755139988834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154298391258624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2763713689088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29879130 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147594283279872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _runtime‚1554.427075846 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69808736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27407340 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149564744071520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2763801718272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚3949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39929604 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2755053840248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154298286190080 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151534589540352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29879130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29879132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152319797911768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17328297 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚154298391258624 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¾möz2Ê - _wandbruntime‚1554  _runtime‚1554„=IRË  /loss/FRt2/detection_loss‚1.3845593393239932 % _timestamp‚1.7844984805169635e+09 1loss/FBt1/assignment_loss‚0.20966514889264196 1loss/FBt2/assignment_loss‚0.01296957824143511 /loss/FBt1/detection_loss‚0.8943913783878099 epoch‚4 _runtime‚1554.427558566 _step‚319 0loss/FRt2/assignment_loss‚1.4203353253647737 2loss/SRqqt1/assignment_loss‚0.8284104212604047 2loss/SRqqt2/assignment_loss‚0.7861384144214552 /loss/FBt2/detection_loss‚1.0882491042106233 &loss/total_loss‚12.089866135879895 trainer/global_step‚3949 /loss/FRt1/assignment_loss‚2.848338165320456 /loss/FRt1/detection_loss‚0.5780779832861559 1loss/SRqqt1/detection_loss‚1.2835280539572134 1loss/SRqqt2/detection_loss‚0.7552032232129325n:éê Ì † 0loss/FRt2/assignment_loss‚1.4203353253647737 2loss/SRqqt1/assignment_loss‚0.8284104212604047 2loss/SRqqt2/assignment_loss‚0.7861384144214552 /loss/FBt2/detection_loss‚1.0882491042106233 &loss/total_loss‚12.089866135879895 trainer/global_step‚3949 /loss/FRt1/assignment_loss‚2.848338165320456 /loss/FRt1/detection_loss‚0.5780779832861559 1loss/SRqqt1/detection_loss‚1.2835280539572134 1loss/SRqqt2/detection_loss‚0.7552032232129325 /loss/FRt2/detection_loss‚1.3845593393239932 % _timestamp‚1.7844984805169635e+09 1loss/FBt1/assignment_loss‚0.20966514889264196 1loss/FBt2/assignment_loss‚0.01296957824143511 /loss/FBt1/detection_loss‚0.8943913783878099 epoch‚4 _runtime‚1554.427558566 _step‚319¿ΛªLÍ ‚(:Á ¶’õÒ›ëé+ disk.nvme0c0n1.in‚ 0.0078125$ memory_percent‚99.19595718383789& proc.memory.availableMB‚ 131.734375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.0078125& disk.nvme1c1n1.out‚4766.2021484375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚4766.2021484375* proc.memory.percent‚17.226672172546387 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 8397.89453125" disk.nvme0n1.out‚ 8397.89453125 disk.nvme1c1n1.in‚0" disk.dm-0.out‚11992.9228515625 disk.dm-1.out‚ 1073.17578125# proc.memory.rssMB‚ 2822.41796875 cpu‚26.185208350057394 network.sent‚ 5.180193e+06 network.recv‚490776) disk./.usagePercent‚75.95491991662723 disk.md0.in‚0 disk.md0.out‚ 45.9140625 disk.dm-0.in‚ 0.0078125 disk.dm-1.in‚0% disk./.usageGB‚1166.0979080200195 disk.rbd0.out‚0Sj¯m›Î ‚(: ¶’õÒžóÑ, gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.311' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.77033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>?u2Ï - _wandbruntime‚1573  _runtime‚1573<èÏ…ÈÐ Â _step‚320 ' lr-AdamW/pg1‚0.00010009962504315815 ' lr-AdamW/pg2‚0.00010009962504315815 trainer/global_step‚3999 % _timestamp‚1.7844984993708942e+09 _runtime‚ 1573.28193874‡öŠÎÍÑ Ç % _timestamp‚1.7844984993708942e+09 _runtime‚ 1573.28193874 _step‚320 ' lr-AdamW/pg1‚0.00010009962504315815 ' lr-AdamW/pg2‚0.00010009962504315815 trainer/global_step‚3999ÀÁð“2Ò - _wandbruntime‚1573  _runtime‚1573óz~°ˆÓ ÎO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1573.287143859 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40406301 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17538504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70644160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30237859 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7844984993729033e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30237861 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156162497816576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2796973321216 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70654305 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27738971 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156162602877952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149377607003648 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17538507 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40416444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27738699 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2878774499840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152q¤éÙ'Ô ÓO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40406301 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17538504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70644160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30237859 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7844984993729033e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30237861 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156162497816576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2796973321216 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70654305 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27738971 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156162602877952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149377607003648 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17538507 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40416444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27738699 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2878774499840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1573.287143859 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3Á:Öš‰2Õ - _wandbruntime‚1573  _runtime‚1573;ìøÞ&Ö ØM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2797651430912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17542794 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2879652409344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149414204974080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153402907901952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70661379 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153402907901952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70671538 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27745528 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7844984996281774e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70661379 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2879474100224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27745793 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70671538 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30245182 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146534552564736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146534527661056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚322 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10202996 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154197333963122 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40426356 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156200647362048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17542797 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154197230775176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153402890862592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151408347459520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2797739460096 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30245180 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40416199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156200542293504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40426356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10202734 _runtime‚1573.539333292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156200647362048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30245180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2797651430912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30245182 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800p’‡¨× ÝM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30245180 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40416199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156200542293504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40426356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10202734 _runtime‚1573.539333292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156200647362048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30245180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2797651430912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30245182 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2797651430912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17542794 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2879652409344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149414204974080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153402907901952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70661379 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153402907901952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70671538 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27745528 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7844984996281774e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70661379 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2879474100224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27745793 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70671538 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30245182 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146534552564736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146534527661056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚322 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10202996 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154197333963122 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40426356 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156200647362048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17542797 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154197230775176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153402890862592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151408347459520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2797739460096 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¸"Ö÷2Ø - _wandbruntime‚1573  _runtime‚1573¶D³<Ù € _step‚323 .loss/FBt2/detection_loss‚0.876033574523076 _runtime‚1573.539823363 2loss/SRqqt1/assignment_loss‚0.7984182247892022 2loss/SRqqt2/assignment_loss‚0.6936821960777386 0loss/SRqqt1/detection_loss‚1.302753980311432 /loss/FBt1/detection_loss‚2.0439014857439015 &loss/total_loss‚12.686493477295377 trainer/global_step‚3999 /loss/FRt2/assignment_loss‚1.178635308392159 1loss/FBt1/assignment_loss‚0.10327311188955453 /loss/FRt2/detection_loss‚1.3842089272318674 1loss/SRqqt2/detection_loss‚0.7243905033936916 epoch‚4 % _timestamp‚1.7844984996291623e+09 0loss/FRt1/assignment_loss‚2.9821795703786793 2loss/FBt2/assignment_loss‚0.011936254822053044 /loss/FRt1/detection_loss‚0.5870803397420231¤àv Ú … 2loss/SRqqt1/assignment_loss‚0.7984182247892022 2loss/SRqqt2/assignment_loss‚0.6936821960777386 0loss/SRqqt1/detection_loss‚1.302753980311432 /loss/FBt1/detection_loss‚2.0439014857439015 &loss/total_loss‚12.686493477295377 trainer/global_step‚3999 /loss/FRt2/assignment_loss‚1.178635308392159 1loss/FBt1/assignment_loss‚0.10327311188955453 /loss/FRt2/detection_loss‚1.3842089272318674 1loss/SRqqt2/detection_loss‚0.7243905033936916 epoch‚4 % _timestamp‚1.7844984996291623e+09 0loss/FRt1/assignment_loss‚2.9821795703786793 2loss/FBt2/assignment_loss‚0.011936254822053044 /loss/FRt1/detection_loss‚0.5870803397420231 _step‚323 .loss/FBt2/detection_loss‚0.876033574523076 _runtime‚1573.539823363Ã=†ëåSÛ ‚(:È Å’õÒبÐ+ disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2822.41796875* proc.memory.percent‚17.226672172546387 proc.cpu.threads‚17" disk.dm-0.out‚12130.2275390625% disk.nvme0c0n1.out‚8484.373046875# disk.nvme0n1.out‚8484.373046875 disk.md0.in‚0$ memory_percent‚99.19638633728027 network.sent‚ 5.229628e+06 network.recv‚495129) disk./.usagePercent‚75.93827687884519" disk.dm-1.out‚1073.71630859375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚4818.90869140625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4818.90869140625 disk.dm-0.in‚ 0.0078125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.0078125 disk.nvme0n1.in‚ 0.0078125 disk.md0.out‚ 46.546875' proc.memory.availableMB‚ 131.6640625 cpu‚26.193698022740758% disk./.usageGB‚1165.8423957824707 disk.dm-1.in‚0/t!ÇÜ ‚(:„ Å’õÒÈý§, gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚191.157' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚63.719 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô(×fGÝ ‚(:¼ Ô’õÒ¹‘´+ disk.dm-0.in‚ 0.0078125 network.recv‚499148! disk.dm-0.out‚13300.365234375# memory_percent‚99.2490291595459 proc.memory.rssMB‚2831 proc.cpu.threads‚17% disk./.usageGB‚1166.2158126831055" disk.dm-1.out‚1073.99755859375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.0078125% disk.nvme0c0n1.out‚9566.212890625 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 123.0390625 network.sent‚ 5.293826e+06 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0' proc.memory.percent‚17.279052734375 cpu‚26.240080945626467# disk.nvme0n1.out‚9566.212890625' disk.nvme1c1n1.out‚4907.98583984375) disk./.usagePercent‚75.96259975138287 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.0078125% disk.nvme1n1.out‚4907.98583984375 disk.md0.out‚ 46.77734375ÔÃ)ËšÞ ‚(: Ô’õÒå¨â+ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.544' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.84800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­É²2ß - _wandbruntime‚1592  _runtime‚15921ñ#¹ˆ=à ‚{ ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13656769692897797 *jet/accuracy_1_of_1‚0.2850024700164795 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7982038855552673 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967535138130188 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13954205811023712 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5569809079170227 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9910317659378052 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8816325068473816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6231493949890137 particle/accuracy_5_of_5‚0 _runtime‚1592.688712575 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972013831138611 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06454545259475708 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06306732445955276 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7552631497383118 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49634602665901184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7392061948776245 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5260255336761475 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7602130174636841 particle/accuracy_2_of_5‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.36391156911849976 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7319176197052002 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3499314785003662 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6167715787887573 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/sensitivity‚0.6248459815979004 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7114034295082092 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1957121193408966 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5041795969009399 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3652859926223755 )jet/accuracy_1_of_3‚0.911553144454956 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12254808843135834 /particle/accuracy_2_of_2‚0.9998950958251953 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26257607340812683 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6090442538261414 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12332043051719666 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.699999988079071 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.597406804561615 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23779596388339996 .particle/accuracy_3_of_3‚0.999514102935791 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13243906199932098 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6154757738113403 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09683576226234436 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49396342039108276 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6574620008468628 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9879972338676453 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967535138130188 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.810240626335144 6validation_average_jet_accuracy‚0.3339497745037079 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16820096969604492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9513459801673889 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5052083730697632 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964715838432312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9830802083015442 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9069070219993591 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5882530808448792 (particle/accuracy‚0.8435291051864624 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5804367661476135 ,jet/accuracy_3_of_3‚0.056504279375076294 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.32716047763824463 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7439254522323608 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7895089387893677 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06514617055654526 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06306732445955276 particle/accuracy_2_of_4‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6654854416847229 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13172072172164917 *jet/accuracy_3_of_4‚0.0842689722776413 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.810240626335144 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008313256315886974 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028090879321098328 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 *jet/accuracy_1_of_2‚0.7636781334877014 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7734116315841675 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11851654946804047 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4004 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.699999988079071 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7483792304992676 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6634249091148376 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6415148973464966 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18854360282421112 *jet/accuracy_2_of_3‚0.5213719010353088 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3055555522441864 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1651025116443634 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6130421161651611 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7034584879875183 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1359064131975174 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8340986371040344 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.997012197971344 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_5‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21893912553787231 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3742271065711975 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5259183049201965 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5520833134651184 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12332043051719666 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6640042662620544 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9868025183677673 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13602674007415771 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946422576904297 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 _step‚324 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10172917693853378 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36683014035224915 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5617428421974182 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1293230503797531 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015334872528910637 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.132131889462471 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.561440646648407 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953502416610718 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.730158805847168 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 /particle/accuracy_1_of_2‚0.9998950958251953 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6715172529220581 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9626666307449341 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8523905873298645 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860550165176392 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.107323557138443 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9979591369628906 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03247128054499626 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2248724102973938 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11837666481733322 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7400140166282654 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015459837391972542 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5000103712081909 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4778573215007782 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.854781448841095 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15722879767417908 +jet/accuracy_4_of_4‚0.01071428507566452 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚11.66805433648427 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6741331219673157 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5343537926673889 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11870215833187103 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12343750149011612 /particle/accuracy_4_of_4‚0.9567286372184753 'particle/f_score‚0.6316472291946411 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06058588996529579 *jet/accuracy_1_of_4‚0.9502533674240112 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6231493949890137 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11590458452701569 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11155100166797638 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.344696968793869 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5177950859069824 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9033246040344238 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7691948413848877 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 % _timestamp‚1.7844985187772515e+09 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.40456345677375793 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.962567925453186 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.455858051776886 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.723668098449707 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5242842435836792 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13582013547420502 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03951476514339447 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11943744122982025 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9534534215927124 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9913728833198547 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7037137150764465 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9836337566375732 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7086870074272156 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06153426319360733 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7444443106651306 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 epoch‚4 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9795104265213013 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6623931527137756,0a†¸á ‡{ 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11155100166797638 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.344696968793869 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5177950859069824 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9033246040344238 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7691948413848877 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 % _timestamp‚1.7844985187772515e+09 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.40456345677375793 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.962567925453186 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.455858051776886 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.723668098449707 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5242842435836792 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13582013547420502 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03951476514339447 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11943744122982025 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9534534215927124 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9913728833198547 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7037137150764465 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9836337566375732 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7086870074272156 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06153426319360733 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7444443106651306 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 epoch‚4 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9795104265213013 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6623931527137756 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13656769692897797 *jet/accuracy_1_of_1‚0.2850024700164795 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7982038855552673 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967535138130188 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13954205811023712 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5569809079170227 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9910317659378052 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8816325068473816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6231493949890137 particle/accuracy_5_of_5‚0 _runtime‚1592.688712575 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972013831138611 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06454545259475708 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06306732445955276 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7552631497383118 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49634602665901184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7392061948776245 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5260255336761475 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7602130174636841 particle/accuracy_2_of_5‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.36391156911849976 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7319176197052002 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3499314785003662 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6167715787887573 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/sensitivity‚0.6248459815979004 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7114034295082092 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1957121193408966 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5041795969009399 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3652859926223755 )jet/accuracy_1_of_3‚0.911553144454956 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12254808843135834 /particle/accuracy_2_of_2‚0.9998950958251953 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26257607340812683 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6090442538261414 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12332043051719666 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.699999988079071 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.597406804561615 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23779596388339996 .particle/accuracy_3_of_3‚0.999514102935791 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13243906199932098 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6154757738113403 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09683576226234436 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49396342039108276 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6574620008468628 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9879972338676453 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967535138130188 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.810240626335144 6validation_average_jet_accuracy‚0.3339497745037079 :"Purity/*FRt2SRqqt0FBt/event_purity‚Ãß Õ!0.16820096969604492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9513459801673889 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5052083730697632 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964715838432312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9830802083015442 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9069070219993591 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5882530808448792 (particle/accuracy‚0.8435291051864624 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5804367661476135 ,jet/accuracy_3_of_3‚0.056504279375076294 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.32716047763824463 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7439254522323608 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7895089387893677 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06514617055654526 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06306732445955276 particle/accuracy_2_of_4‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6654854416847229 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13172072172164917 *jet/accuracy_3_of_4‚0.0842689722776413 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.810240626335144 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008313256315886974 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028090879321098328 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 *jet/accuracy_1_of_2‚0.7636781334877014 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7734116315841675 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11851654946804047 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4004 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.699999988079071 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7483792304992676 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6634249091148376 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6415148973464966 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18854360282421112 *jet/accuracy_2_of_3‚0.5213719010353088 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3055555522441864 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1651025116443634 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6130421161651611 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7034584879875183 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1359064131975174 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8340986371040344 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.997012197971344 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_5‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21893912553787231 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3742271065711975 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5259183049201965 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5520833134651184 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12332043051719666 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6640042662620544 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9868025183677673 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13602674007415771 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9946422576904297 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 _step‚324 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10172917693853378 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36683014035224915 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5617428421974182 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1293230503797531 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015334872528910637 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.132131889462471 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.561440646648407 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953502416610718 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.730158805847168 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 /particle/accuracy_1_of_2‚0.9998950958251953 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6715172529220581 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9626666307449341 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8523905873298645 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860550165176392 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.107323557138443 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9979591369628906 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03247128054499626 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2248724102973938 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11837666481733322 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7400140166282654 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015459837391972542 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5000103712081909 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4778573215007782 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.854781448841095 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15722879767417908 +jet/accuracy_4_of_4‚0.01071428507566452 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚11.66805433648427 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6741331219673157 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5343537926673889 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11870215833187103 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12343750149011612 /particle/accuracy_4_of_4‚0.9567286372184753 'particle/f_score‚0.6316472291946411 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06058588996529579 *jet/accuracy_1_of_4‚0.9502533674240112 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6231493949890137 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11590458452701569Ä,p°Sâ ‚(:È ã’õÒ¹äÕ+% disk./.usageGB‚1166.2163925170898 disk.nvme0c0n1.in‚ 0.0078125& disk.nvme0c0n1.out‚9567.0146484375' disk.nvme1c1n1.out‚4972.21044921875 disk.dm-0.in‚ 0.0078125 disk.nvme1n1.in‚0 disk.dm-1.in‚0# memory_percent‚98.8046407699585 cpu‚26.251751603124678 disk.md0.in‚0$ disk.nvme0n1.out‚9567.0146484375% disk.nvme1n1.out‚4972.21044921875! disk.dm-0.out‚13300.994140625" disk.dm-1.out‚1137.93994140625 proc.cpu.threads‚17 network.recv‚502628 disk.nvme0n1.in‚ 0.0078125 disk.nvme1c1n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 195.84765625" proc.memory.rssMB‚ 2831.6484375* proc.memory.percent‚17.283010482788086 disk.md0.out‚ 46.984375 network.sent‚ 5.309591e+06) disk./.usagePercent‚75.96263751943265ì4b±ã ‚(:„ ã’õÒ–Ø…, gpu.0.gpu‚52.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚150.072' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚50.024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=êË2ä - _wandbruntime‚1611  _runtime‚1611|£Éå à trainer/global_step‚4049 % _timestamp‚1.7844985372526994e+09 _runtime‚1611.164268411 _step‚325 ' lr-AdamW/pg1‚0.00010004767561757339 ' lr-AdamW/pg2‚0.00010004767561757339íº¹)Îæ È ' lr-AdamW/pg1‚0.00010004767561757339 ' lr-AdamW/pg2‚0.00010004767561757339 trainer/global_step‚4049 % _timestamp‚1.7844985372526994e+09 _runtime‚1611.164268411 _step‚325ÅbJs2ç - _wandbruntime‚1611  _runtime‚1611Á|.Ô'è ÎO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2851331380224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41070319 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157026102283132 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2851419402240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7844985372533953e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71911696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152143232065024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159057942893568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156206506451968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28250748 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149209013629952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159057837832192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2851331380224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149209038533632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30841379 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157025999095298 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71921841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚326 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152143433187840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41080462 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156206523491328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156206523491328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10403897 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2934218435072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159057837832192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41070319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28250984 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1611.164328591 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156206506451968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159057942893568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10403664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154183585287472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2842430847186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2851419402240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30841377 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30841377 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41080462 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17847084 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30841379 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2842516995660 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1611.164328591 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156206506451968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159057942893568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10403664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154183585287472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2842430847186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2851419402240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30841377 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30841377 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41080462 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17847084 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30841379 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2842516995660 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159057837832192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2851331380224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149209038533632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30841379 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157025999095298 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71921841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚326 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152143433187840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41080462 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156206523491328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156206523491328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10403897 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2934218435072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159057837832192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41070319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28250984 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28257540 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_byte­5»s.all.allocated‚152179835218944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41090374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156243889858560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30848700 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41080217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2852009489920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚327 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156243872819200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41080217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2843106798630 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159095882309120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71928915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159095987377664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10406166 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152179621520384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159095987377664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157063547114502 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71939074 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30848698 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2935101528064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2843192947216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71939074 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30848700 _runtime‚1611.430385009 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2852009489920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28257789 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2934912733184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157063650302448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 $ _timestamp‚1.784498537518647e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30848698 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10406412 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712䬡$â&ì ÜM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17851377 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚154220440315872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28257540 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152179835218944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41090374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156243889858560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30848700 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41080217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2852009489920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚327 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156243872819200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41080217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2843106798630 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159095882309120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71928915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159095987377664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10406166 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152179621520384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159095987377664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157063547114502 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71939074 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30848698 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2935101528064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2843192947216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71939074 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30848700 _runtime‚1611.430385009 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2852009489920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28257789 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2934912733184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157063650302448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 $ _timestamp‚1.784498537518647e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30848698 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10406412 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149244708787200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154220457355232 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816ÇUå2í - _wandbruntime‚1611  _runtime‚1611_õ*.î € 2loss/SRqqt2/assignment_loss‚0.6719788900442673 /loss/FRt1/detection_loss‚0.6456684302149406 /loss/FRt2/detection_loss‚1.4541764218975584 0loss/FBt1/assignment_loss‚0.3254359638882022 epoch‚5 trainer/global_step‚4049 _runtime‚1611.430596209 /loss/FBt2/detection_loss‚1.0686287977173925 .loss/FRt1/assignment_loss‚2.71040184208841 3loss/FBt2/assignment_loss‚0.0013611931299146818 1loss/SRqqt1/detection_loss‚1.2459203783886295 .loss/FBt1/detection_loss‚1.497093263391332 % _timestamp‚1.7844985375196013e+09 0loss/FRt2/assignment_loss‚1.3545340889334256 1loss/SRqqt2/detection_loss‚0.8869209559100313 _step‚328 &loss/total_loss‚12.581024271662436 2loss/SRqqt1/assignment_loss‚0.7189040460583308“çÛÉ ï … &loss/total_loss‚12.581024271662436 2loss/SRqqt1/assignment_loss‚0.7189040460583308 2loss/SRqqt2/assignment_loss‚0.6719788900442673 /loss/FRt1/detection_loss‚0.6456684302149406 /loss/FRt2/detection_loss‚1.4541764218975584 0loss/FBt1/assignment_loss‚0.3254359638882022 epoch‚5 trainer/global_step‚4049 _runtime‚1611.430596209 /loss/FBt2/detection_loss‚1.0686287977173925 .loss/FRt1/assignment_loss‚2.71040184208841 3loss/FBt2/assignment_loss‚0.0013611931299146818 1loss/SRqqt1/detection_loss‚1.2459203783886295 .loss/FBt1/detection_loss‚1.497093263391332 % _timestamp‚1.7844985375196013e+09 0loss/FRt2/assignment_loss‚1.3545340889334256 1loss/SRqqt2/detection_loss‚0.8869209559100313 _step‚328È„@Zð ‚(:Ï ò’õÒ›ýÆ+ disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4992.80615234375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4992.80615234375( proc.memory.availableMB‚ 195.81640625" proc.memory.rssMB‚ 2831.6484375 network.sent‚ 5.366377e+06 network.recv‚506590 disk.dm-1.in‚0" disk.dm-1.out‚1154.11962890625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.01171875$ memory_percent‚98.80483150482178) disk./.usagePercent‚75.96243551006108$ disk./.usageGB‚1166.213291168213 disk.md0.in‚0 disk.dm-0.in‚ 0.01171875# disk.dm-0.out‚13330.06298828125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.01171875 proc.cpu.threads‚17 disk.md0.out‚ 47.328125' disk.nvme0c0n1.out‚9592.41357421875% disk.nvme0n1.out‚9592.41357421875* proc.memory.percent‚17.283010482788086 cpu‚26.259699726661506¹Ù¯šñ ‚(: ò’õÒâÃï+ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.806' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.26866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬gõ2ò - _wandbruntime‚1630  _runtime‚1630åY’ÖÇó Á & lr-AdamW/pg1‚9.999493865126803e-05 & lr-AdamW/pg2‚9.999493865126803e-05 trainer/global_step‚4099 % _timestamp‚1.7844985570094485e+09 _runtime‚1630.920481337 _step‚329ÔÙÃÏÌô Æ & lr-AdamW/pg1‚9.999493865126803e-05 & lr-AdamW/pg2‚9.999493865126803e-05 trainer/global_step‚4099 % _timestamp‚1.7844985570094485e+09 _runtime‚1630.920481337 _step‚329ÉW¦Ÿ2õ - _wandbruntime‚1630  _runtime‚1630¦ÐŒÔ'ö ÎO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31207429 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31207427 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2885269122048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160960093935616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156027188675472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160960093935616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158074824813568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41577214 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10527120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72774498 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153962941610496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41567071 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10526878 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2885357144064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2876260322594 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2885357144064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844985570101893e+09 _runtime‚1630.921034498 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28588462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31207429 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160960198996992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156027171636112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41567071 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28588707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18061584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150993771492352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41577214 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158903535146540 _step‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160960198996992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158903431958706 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2969170118144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158074841852928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18061587³,`á÷ ÓO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158074841852928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18061587 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31207429 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.Çð¾†Ó#large_pool.freed‚31207427 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2885269122048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160960093935616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156027188675472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160960093935616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158074824813568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41577214 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10527120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72774498 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153962941610496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41567071 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10526878 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2885357144064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2876260322594 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2885357144064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844985570101893e+09 _runtime‚1630.921034498 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28588462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31207429 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160960198996992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156027171636112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41567071 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28588707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18061584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150993771492352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41577214 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158903535146540 _step‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160960198996992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158903431958706 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2969170118144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2ÊY³t2ø - _wandbruntime‚1631  _runtime‚1631ÃèÒ Þ&ù ØM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31214750 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10529493 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72791717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160998243481088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚331 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2877022422624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31214750 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160998243481088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28595127 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31214748 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1631.174030596 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160998138412544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158112208220160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153999539343360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72801876 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31214748 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2970047790080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41576969 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10529253 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2969865286656 % _timestamp‚1.7844985572629347e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41576969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72801876 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41587126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ÏûÐ.ã&ú ÝM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2969865286656 % _timestamp‚1.7844985572629347e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41576969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72801876 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41587126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31214750 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10529493 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72791717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160998243481088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚331 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2877022422624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31214750 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160998243481088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28595127 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31214748 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1631.174030596 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160998138412544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158112208220160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153999539343360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72801876 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31214748 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2970047790080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41576969 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10529253 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Ëq.*>2û - _wandbruntime‚1631  _runtime‚1631˜6¶ü € /loss/FBt1/detection_loss‚1.6183138752684874 &loss/total_loss‚12.072179625030891 /loss/FRt1/assignment_loss‚2.716258060328568 1loss/SRqqt1/detection_loss‚1.2604451855711107 trainer/global_step‚4099 _runtime‚1631.174629617 /loss/FRt2/assignment_loss‚1.011514048081524 2loss/SRqqt2/assignment_loss‚0.6927980472994465 1loss/FBt1/assignment_loss‚0.11585980703290154 1loss/FBt2/assignment_loss‚0.10115217467441359 epoch‚5 % _timestamp‚1.7844985572638867e+09 2loss/SRqqt1/assignment_loss‚0.8790952310636861 /loss/FRt1/detection_loss‚0.6288970593143356 /loss/FRt2/detection_loss‚1.4695034300416427 1loss/SRqqt2/detection_loss‚0.7131754886141878 /loss/FBt2/detection_loss‚0.8651672177405881 _step‚3325þò¶ ý … 1loss/FBt2/assignment_loss‚0.10115217467441359 epoch‚5 % _timestamp‚1.7844985572638867e+09 2loss/SRqqt1/assignment_loss‚0.8790952310636861 /loss/FRt1/detection_loss‚0.6288970593143356 /loss/FRt2/detection_loss‚1.4695034300416427 1loss/SRqqt2/detection_loss‚0.7131754886141878 /loss/FBt2/detection_loss‚0.8651672177405881 _step‚332 /loss/FBt1/detection_loss‚1.6183138752684874 &loss/total_loss‚12.072179625030891 /loss/FRt1/assignment_loss‚2.716258060328568 1loss/SRqqt1/detection_loss‚1.2604451855711107 trainer/global_step‚4099 _runtime‚1631.174629617 /loss/FRt2/assignment_loss‚1.011514048081524 2loss/SRqqt2/assignment_loss‚0.6927980472994465 1loss/FBt1/assignment_loss‚0.11585980703290154ÌGNf[þ ‚(:Ð “õÒ²à¹, disk.md0.out‚ 47.7578125$ memory_percent‚98.80499839782715" proc.memory.rssMB‚ 2831.6484375 disk.dm-0.in‚ 0.01171875" disk.dm-1.out‚1202.12548828125 disk.nvme0n1.in‚ 0.01171875% disk.nvme0n1.out‚9593.46337890625 disk.nvme1n1.in‚0 disk.md0.in‚0 proc.cpu.threads‚17 network.sent‚ 5.415725e+06% disk./.usageGB‚1166.2139320373535 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5041.33837890625% disk.nvme1n1.out‚5041.33837890625 cpu‚26.263222630243344 network.recv‚511122) disk./.usagePercent‚75.96247725369507# disk.dm-0.out‚13330.70361328125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.01171875' disk.nvme0c0n1.out‚9593.46337890625' proc.memory.availableMB‚ 195.7890625* proc.memory.percent‚17.283010482788086 disk.rbd0.in‚0•bž)šÿ ‚(: “õÒï™÷, gpu.0.gpu‚78.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚113.029' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.67633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Vý[2€- _wandbruntime‚1649  _runtime‚1649ˆ\³ÆÀ trainer/global_step‚4149 $ _timestamp‚1.784498575663904e+09 _runtime‚1649.576421516 _step‚333 & lr-AdamW/pg1‚9.994141498893164e-05 & lr-AdamW/pg2‚9.994141498893164e-05ª,!àË‚Å & lr-AdamW/pg1‚9.994141498893164e-05 & lr-AdamW/pg2‚9.994141498893164e-05 trainer/global_step‚4149 $ _timestamp‚1.784498575663904e+09 _runtime‚1649.576421516 _step‚333Íå!d2ƒ- _wandbruntime‚1649  _runtime‚16492ly«„ÎO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42063823 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 Z0Bê:)'FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155782648991744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152778554258432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159943143175168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚225 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2919206863872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3004119636992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162862455100416 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3004289564672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157870792063472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28926195 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28925970 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18276084 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155782843823104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162862350039040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10650108 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73637300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910089798002 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844985756645079e+09 _runtime‚1649.576503586 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42073966 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162862455100416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160780968009948 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31573479 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159943160214528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159943143175168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10649886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42063823 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2919206863872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31573477 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31573477 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18276087 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152778529354752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160780864822114 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73647445 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2910175946476 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2919294885888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42073966 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31573479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚4149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18276084 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚222 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155782843823104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162862350039040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10650108 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73637300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910089798002 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844985756645079e+09 _runtime‚1649.576503586 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42073966 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162862455100416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160780968009948 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31573479 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159943160214528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159943143175168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10649886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42063823 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2919206863872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31573477 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31573477 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18276087 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152778529354752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160780864822114 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73647445 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2910175946476 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2919294885888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42073966 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31573479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚4149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3004811614720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160818516029264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73654519 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73654519 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31580798 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2910765749446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2919973002752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159980526581760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42073721 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2919884973568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31580798 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28932937 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚335 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160818412841318 _runtime‚1649.829026966 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18280377 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73664678 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28932691 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2910851898032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155819036126720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18280377 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73664678 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28932691 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2910851898032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155819036126720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3004811614720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160818516029264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73654519 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73654519 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31580798 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2910765749446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2919973002752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159980526581760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42073721 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2919884973568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31580798 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28932937 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚335 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160818412841318 _runtime‚1649.829026966ÏœBu2‰- _wandbruntime‚1649  _runtime‚1649…ôJ‹Šÿ 0loss/FRt1/detection_loss‚0.49893410433434254 1loss/SRqqt2/detection_loss‚0.6477496149531423 epoch‚5 trainer/global_step‚4149 2loss/SRqqt1/assignment_loss‚0.8223758410282985 1loss/FBt1/assignment_loss‚0.31395098572622476 2loss/FBt2/assignment_loss‚0.009744378294805867 0loss/SRqqt1/detection_loss‚1.253740588430729 /loss/FBt1/detection_loss‚1.8765497730480722 % _timestamp‚1.7844985759188788e+09 0loss/FRt1/assignment_loss‚2.8261217571114785 .loss/FRt2/assignment_loss‚1.44649255715613 2loss/SRqqt2/assignment_loss‚0.7312923462454383 /loss/FRt2/detection_loss‚1.4153012402554994 %loss/total_loss‚12.69873636688582 _runtime‚1649.829537797 _step‚336 .loss/FBt2/detection_loss‚0.856483180301659£¯v ‹„ _runtime‚1649.829537797 _step‚336 .loss/FBt2/detection_loss‚0.856483180301659 0loss/FRt1/detection_loss‚0.49893410433434254 1loss/SRqqt2/detection_loss‚0.6477496149531423 epoch‚5 trainer/global_step‚4149 2loss/SRqqt1/assignment_loss‚0.8223758410282985 1loss/FBt1/assignment_loss‚0.31395098572622476 2loss/FBt2/assignment_loss‚0.009744378294805867 0loss/SRqqt1/detection_loss‚1.253740588430729 /loss/FBt1/detection_loss‚1.8765497730480722 % _timestamp‚1.7844985759188788e+09 0loss/FRt1/assignment_loss‚2.8261217571114785 .loss/FRt2/assignment_loss‚1.44649255715613 2loss/SRqqt2/assignment_loss‚0.7312923462454383 /loss/FRt2/detection_loss‚1.4153012402554994 %loss/total_loss‚12.69873636688582ÐX“TŒ‚(:É “õÒü‚Î+* proc.memory.percent‚17.283010482788086 network.recv‚515697 disk.nvme0n1.in‚ 0.01171875% disk.nvme0n1.out‚9627.48974609375% disk.nvme1n1.out‚5042.03271484375 disk.md0.out‚ 48.04296875# disk.dm-0.out‚13364.70751953125# memory_percent‚98.8067626953125 cpu‚26.260725601843202) disk./.usagePercent‚75.98212806938953' disk.nvme0c0n1.out‚9627.48974609375' disk.nvme1c1n1.out‚5042.03271484375 disk.nvme1n1.in‚0 disk.md0.in‚0 network.sent‚ 5.49107e+06 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.01171875 disk.dm-1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1166.5156211853027 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.01171875" disk.dm-1.out‚1202.21923828125! proc.memory.availableMB‚195.5" proc.memory.rssMB‚ 2831.6484375ÁV‚@‚(:… “õÒçëù+ gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.584' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nß©SŽ‚(:È Ÿ“õÒ¼¹é+$ memory_percent‚98.80542755126953 disk.dm-1.in‚0% disk.nvme1n1.out‚5042.99267578125 disk.md0.in‚0% proc.memory.availableMB‚ 195.71875 cpu‚26.26553890994539 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.01171875% disk.nvme0n1.out‚9656.76318359375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.01171875 network.recv‚520343 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.01171875' disk.nvme1c1n1.out‚5042.99267578125 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 2831.6484375* proc.memory.percent‚17.283010482788086 network.sent‚ 5.542294e+06) disk./.usagePercent‚75.96696966382778$ disk./.usageGB‚1166.282901763916" disk.dm-1.out‚1202.51513671875' disk.nvme0c0n1.out‚9656.76318359375 disk.md0.out‚48.625# disk.dm-0.out‚13393.39501953125p­:Û‚(:„ Ÿ“õÒÀÍ“, gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚184.392' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚61.464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é°2- _wandbruntime‚1669  _runtime‚1669žø–îÇ‘Á & lr-AdamW/pg1‚9.988710548785428e-05 & lr-AdamW/pg2‚9.988710548785428e-05 trainer/global_step‚4199 % _timestamp‚1.7844985953897953e+09 _runtime‚1669.300745757 _step‚337‡FÎsÌ’Æ & lr-AdamW/pg2‚9.988710548785428e-05 trainer/global_step‚4199 % _timestamp‚1.7844985953897953e+09 _runtime‚1669.300745757 _step‚337 & lr-AdamW/pg1‚9.988710548785428e-05Ñgž)_2“- _wandbruntime‚1669  _runtime‚1669è«B-Ó'”ÍO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18490584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10773282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2953232627712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3039232799744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164764606142464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161811461536768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74510247 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚4199 _step‚338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161811478576128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18490587 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2944005421884 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157602544920576 _runtime‚ 1669.30172395 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161811461536768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2953144605696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31939527 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159714395451472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29263616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10773032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2943919273410 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42570718 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159714378412112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161811478576128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162658297685522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164764711203840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2953232627712·›¢8Ø'•ÒO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18490587 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2944005421884 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157602544920576 _runtime‚ 1669.30172395 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161811461536768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2953144605696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31939527 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159714395451472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29263616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10773032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2943919273410 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42570718 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159714378412112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161811478576128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162658297685522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164764711203840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2953232627712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18490584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10773282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2953232627712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3039232799744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164764606142464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161811461536768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74510247 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚4199 _step‚338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161811478576128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208Òû‚€2–- _wandbruntime‚1669  _runtime‚1669ž·jŸ—×M UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74517Ž|ú>&321 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42580630 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2944681373440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3039748814336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162695948892672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 $ _timestamp‚1.784498595656718e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42570473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2953910744576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164802755687936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159751267519232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10775769 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161848844943360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18494877 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3039922929152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31946848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29270385 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 _step‚339 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164802755687936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31946850 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74527480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2953910744576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2953822715392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74517321 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161848827904000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42570473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18494874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29270646 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74527480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31946848 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157638930207232;ÒÐTâ&˜ÜM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3039748814336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162695948892672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 $ _timestamp‚1.784498595656718e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42570473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2953910744576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164802755687936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159751267519232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10775769 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161848844943360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18494877 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3039922929152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31946848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29270385 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 _step‚339 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164802755687936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31946850 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74527480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2953910744576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2953822715392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74517321 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161848827904000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42570473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18494874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29270646 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74527480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31946848 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157638930207232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74517321 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42580630 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2944681373440 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43057327 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32305577 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164535730548930 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32305579 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18705087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2987170369536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 1689.48334268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164535833736764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29601219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2977834897292 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚163679779898368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚166666862245888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75373049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3073975376896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2977748748818 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚342 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43067470 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43057327 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166666967307264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚163679796937728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161557981800112 % _timestamp‚1.7844986155725029e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166666862245888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32305579 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29600984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10895900 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43067470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚4249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166666967307264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10896132 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161557998839472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2987082347520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156348045079552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3074164178944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2987082347520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2987170369536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43057327 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32305577 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164535730548930 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32305579 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18705087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2987170369536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 1689.48334268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164535833736764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29601219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2977834897292 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚163679779898368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚166666862245888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75373049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3073975376896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2977748748818 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚342 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43067470 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43057327 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166666967307264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚163679796937728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161557981800112 % _timestamp‚1.7844986155725029e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166666862245888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32305579 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29600984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10895900 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43067470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚4249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166666967307264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10896132 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161557998839472Öd¾z2¤- _wandbruntime‚1689  _runtime‚1689áqºÞ&¥ØM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166705011791360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1689.752016413 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75380123 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163717146265600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75380123 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161594853867872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32312900 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75390282 trainer/global_step‚4249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3074675512832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166705011791360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32312898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43077382 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2978424700262 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43067225 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32312900 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29607693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10898319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2987848486400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32312898 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164573381756080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75390282 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163717163304960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2987760457216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10898556 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163717163304960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43067225 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18709374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159458623156736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2978510848848 % _timestamp‚1.7844986158406727e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2987848486400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164573278568134 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156383740236800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159458415749632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2987760457216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161594870907232¼=3rã&¦ÝM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75390282 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163717163304960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2987760457216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10898556 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163717163304960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43067225 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18709374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159458623156736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2978510848848 % _timestamp‚1.7844986158406727e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2987848486400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164573278568134 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156383740236800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159458415749632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2987760457216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161594870907232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166705011791360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1689.752016413 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75380123 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163717146265600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75380123 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161594853867872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32312900 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75390282 trainer/global_step‚4249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3074675512832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166705011791360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32312898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43077382 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2978424700262 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43067225 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32312900 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29607693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10898319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2987848486400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32312898 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164573381756080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0×k'02§- _wandbruntime‚1689  _runtime‚1689¢™‚6¨€ 1loss/SRqqt1/assignment_loss‚0.852330819970709 0loss/FBt1/assignment_loss‚0.2648960630153695 /loss/FRt2/detection_loss‚1.4980687844371903 % _timestamp‚1.7844986158423982e+09 _runtime‚1689.753156245 0loss/FRt1/assignment_loss‚2.7740744065651692 &loss/total_loss‚12.026674560368619 trainer/global_step‚4249 1loss/SRqqt1/detection_loss‚1.3905729005239076 1loss/SRqqt2/detection_loss‚0.8011683832924076 /loss/FBt1/detection_loss‚0.9736891152529883 _step‚344 0loss/FRt2/assignment_loss‚1.1447848484450178 1loss/SRqqt2/assignment_loss‚0.804952822138087 2loss/FBt2/assignment_loss‚0.015480956726147611 /loss/FRt1/detection_loss‚0.6003522868853524 /loss/FBt2/detection_loss‚0.9063031731162728 epoch‚5?„ ©… /loss/FRt2/detection_loss‚1.4980687844371903 % _timestamp‚1.7844986158423982e+09 _runtime‚1689.753156245 0loss/FRt1/assignment_loss‚2.7740744065651692 &loss/total_loss‚12.026674560368619 trainer/global_step‚4249 1loss/SRqqt1/detection_loss‚1.3905729005239076 1loss/SRqqt2/detection_loss‚0.8011683832924076 /loss/FBt1/detection_loss‚0.9736891152529883 _step‚344 0loss/FRt2/assignment_loss‚1.1447848484450178 1loss/SRqqt2/assignment_loss‚0.804952822138087 2loss/FBt2/assignment_loss‚0.015480956726147611 /loss/FRt1/detection_loss‚0.6003522868853524 /loss/FBt2/detection_loss‚0.9063031731162728 epoch‚5 1loss/SRqqt1/assignment_loss‚0.852330819970709 0loss/FBt1/assignment_loss‚0.2648960630153695Ø4±ØTª‚(:É ½“õÒù΃, disk.md0.out‚ 49.2890625 disk.dm-0.in‚ 0.01171875 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚9812.97998046875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 195.65234375 network.sent‚ 5.62022e+06 network.recv‚528036) disk./.usagePercent‚75.96ª e278387062559 disk.md0.in‚0 disk.nvme0c0n1.in‚ 0.01171875# disk.nvme1n1.out‚5105.525390625" disk.dm-1.out‚1202.76708984375 disk.nvme0n1.in‚ 0.01171875% disk.nvme0n1.out‚9812.97998046875" memory_percent‚98.805832862854* proc.memory.percent‚17.283010482788086% disk./.usageGB‚1166.2186393737793" disk.dm-0.out‚13610.4404296875 disk.rbd0.in‚0% disk.nvme1c1n1.out‚5105.525390625" proc.memory.rssMB‚ 2831.6484375 cpu‚26.282529982314635 proc.cpu.threads‚17ðPbš«‚(: ½“õҜر, gpu.0.gpu‚69.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.576' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.52533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óü˜ 2¬- _wandbruntime‚1709  _runtime‚1709מÅ­¿ _step‚345 & lr-AdamW/pg1‚9.977613246155715e-05 & lr-AdamW/pg2‚9.977613246155715e-05 trainer/global_step‚4299 # _timestamp‚1.78449863549688e+09 _runtime‚1709.407894371•ë¡¥Ê®Ä & lr-AdamW/pg1‚9.977613246155715e-05 & lr-AdamW/pg2‚9.977613246155715e-05 trainer/global_step‚4299 # _timestamp‚1.78449863549688e+09 _runtime‚1709.407894371 _step‚345Ù‡‹j2¯- _wandbruntime‚1709  _runtime‚1709Ÿ¿¾Ô'°ÎO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32671627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32671629 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11018889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158132827845632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3109137310208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3011578224226 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168569118349312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3108958993920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161241761935872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76235851 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3011664372700 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43554079 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32671627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3021108111360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29938476 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18919584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168569223410688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3021020089344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚4299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3021108111360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168569118349312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165548098259968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18919587 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166413163412338 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29938235 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163401602227472 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165548115299328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32671629 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚1709.413423151 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166413266600172 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43564222 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43564222 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161241965155840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3021020089344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168569223410688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165548115299328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11018651 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362,l¦Ù'±ÓO % _timestamp‚1.7844986354990048e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76225706 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158132802941952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43554079 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165548115299328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32671629 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚1709.413423151 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166413266600172 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43564222 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43564222 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161241965155840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3021020089344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168569223410688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165548115299328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11018651 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32671627 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32671629 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11018889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158132827845632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3109137310208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3011578224226 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168569118349312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3108958993920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161241761935872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76235851 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3011664372700 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43554079 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32671627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3021108111360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29938476 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18919584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168569223410688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3021020089344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚4299 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3021108111360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168569118349312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165548098259968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18919587 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166413163412338 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29938235 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163401602227472 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163438457255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43563977 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3021698199040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76242925 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43574134 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29945035 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3021786228224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158168498099200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166450711431542 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18923877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168607267894784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165585464627200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3109843930112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3021786228224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158168523002880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3109657232384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161278366932992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166450814619488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32678950 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32678948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11021161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18923874 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76253084 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32678948 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165585464627200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 1709.99267854 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29945288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3012340324256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3012254175670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482ãtÛƒk´ÛM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165585464627200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3109843930112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3021786228224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158168523002880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3109657232384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161278366932992 JDDeviceStatsMonitor.on_train_batch_end/oversize_alloca~¡}v#tions.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166450814619488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32678950 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32678948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11021161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18923874 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76253084 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32678948 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165585464627200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 1709.99267854 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29945288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3012340324256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3012254175670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163438457255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43563977 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3021698199040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76242925 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43574134 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29945035 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3021786228224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158168498099200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166450711431542 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18923877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168607267894784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ûƒb˜–2µ- _wandbruntime‚1709  _runtime‚1709CPÔÛ¶€ 1loss/FBt1/assignment_loss‚0.22953909181455856 /loss/FBt1/detection_loss‚1.8673498330172151 &loss/total_loss‚12.281051559190722 epoch‚5 .loss/FRt2/detection_loss‚1.470919321736037 /loss/FBt2/detection_loss‚0.7619417396560312 _runtime‚1709.997371318 /loss/FRt1/detection_loss‚0.5812582818397897 trainer/global_step‚4299 % _timestamp‚1.7844986360830188e+09 _step‚348 2loss/SRqqt2/assignment_loss‚0.7004975811310151 2loss/FBt2/assignment_loss‚0.007825543671083323 1loss/SRqqt1/detection_loss‚1.4356996006360139 1loss/SRqqt2/detection_loss‚0.6796402444377913 /loss/FRt1/assignment_loss‚2.855789516322299 /loss/FRt2/assignment_loss‚1.125812092899833 2loss/SRqqt1/assignment_loss‚0.5647787120290545ìÔ{ ·… /loss/FRt1/detection_loss‚0.5812582818397897 trainer/global_step‚4299 % _timestamp‚1.7844986360830188e+09 _step‚348 2loss/SRqqt2/assignment_loss‚0.7004975811310151 2loss/FBt2/assignment_loss‚0.007825543671083323 1loss/SRqqt1/detection_loss‚1.4356996006360139 1loss/SRqqt2/detection_loss‚0.6796402444377913 /loss/FRt1/assignment_loss‚2.855789516322299 /loss/FRt2/assignment_loss‚1.125812092899833 2loss/SRqqt1/assignment_loss‚0.5647787120290545 1loss/FBt1/assignment_loss‚0.22953909181455856 /loss/FBt1/detection_loss‚1.8673498330172151 &loss/total_loss‚12.281051559190722 epoch‚5 .loss/FRt2/detection_loss‚1.470919321736037 /loss/FBt2/detection_loss‚0.7619417396560312 _runtime‚1709.997371318ÜÄ–ÍQ¸‚(:Æ Ì“õÒÒ׫+% disk./.usageGB‚1166.2189140319824 disk.nvme0c0n1.in‚ 0.01171875& disk.nvme0c0n1.out‚9814.2275390625 disk.nvme1c1n1.in‚0" disk.dm-1.out‚1202.85302734375 disk.rbd0.out‚0# memory_percent‚99.2023229598999* proc.memory.percent‚17.283010482788086 proc.cpu.threads‚17$ disk.nvme0n1.out‚9814.2275390625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.01171875 disk.dm-1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 130.69140625" proc.memory.rssMB‚ 2831.6484375 network.sent‚ 5.668647e+06 disk.nvme0n1.in‚ 0.01171875' disk.nvme1c1n1.out‚5106.17431640625 disk.md0.in‚0 disk.dm-0.out‚ 13611.6875 cpu‚26.289813326573178% disk.nvme1n1.out‚5106.17431640625 disk.md0.out‚ 49.546875 network.recv‚531535) disk./.usagePercent‚75.96280176075443§„7åš¹‚(: Ì“õÒ–®«, gpu.0.gpu‚45.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.475' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.15833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w“ëØMº‚(: ۓõÒ§€ä+) disk./.usagePercent‚75.96280176075443 disk.dm-0.in‚ 0.01171875 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1166.2189140319824 disk.nvme0c0n1.in‚ 0.01171875& disk.nvme0c0n1.out‚9815.2958984375 disk.nvme0n1.in‚ 0.01171875$ disk.nvme0n1.out‚9815.2958984375& disk.nvme1c1n1.out‚5106.9287109375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚5106.9287109375 network.sent‚ 5.732714e+06 disk.md0.out‚ 50.0546875 disk.dm-0.out‚ 13612.2578125 disk.dm-1.out‚ 1203.015625 disk.rbd0.out‚0$ memory_percent‚99.20110702514648" proc.memory.rssMB‚ 2831.6484375 cpu‚26.297079024196712 network.recv‚536178 disk.nvme1c1n1.in‚0 disk.md0.in‚0& proc.memory.availableMB‚ 130.890625* proc.memory.percent‚17.283010482788086(+b»‚(:„ Û“õÒ阒, gpu.0.gpu‚52.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚139.524' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚46.508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qCÕn2¼- _wandbruntime‚1729  _runtime‚1729Z·Æ½À _step‚349 & lr-AdamW/pg1‚9.971947071379563e-05 & lr-AdamW/pg2‚9.971947071379563e-05 trainer/global_step‚4349 % _timestamp‚1.7844986557417972e+09 _runtime‚ 1729.65271389[ߠ˾Š_runtime‚ 1729.65271389 _step‚349 & lr-AdamW/pg1‚9.971947071379563e-05 & lr-AdamW/pg2‚9.971947071379563e-05 trainer/global_step‚4349 % _timestamp‚1.7844986557417972e+09Ý4L$2¿- _wandbruntime‚1729  _runtime‚1729²B|Ô'ÀÎO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170471374452736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33037679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19134084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19134087 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165245205615472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7844986557428424e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚350 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170471479514112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163061456035840 trainer/global_step‚4349 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77088508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _runtime‚1729.653799091 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167416416621568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44060974 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165245188576112 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44050831 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3144065159168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159917585708032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3055045853184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168290699463580 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44060974 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170471479514112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170471374452736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11141866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159917560804352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44050831 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167416433660928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167416433660928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3055045853184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482gâ-‡U"ÁÓO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159917585708032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3055045853184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168290699463580 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44060974 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170471479514112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170471374452736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11141866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159917560804352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44050831 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167416433660928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167416433660928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3055045853184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170471374452736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33037679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19134084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19134087 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165245205615472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7844986557428424e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚350 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170471479514112 ZFDeviceStats óH±„Monitor.on_train_batch_start/inactive_split_bytes.all.freed‚163061456035840 trainer/global_step‚4349 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77088508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _runtime‚1729.653799091 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167416416621568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44060974 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165245188576112 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44050831 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3144065159168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360ÞvÙÝ'2Â- _wandbruntime‚1729  _runtime‚1729ȃë¼Ý&Ã×M \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3055723970048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19138377 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44070886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170509523998208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚351 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167453782988800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167453800028160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3144771731968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3046083651078 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚4349 _runtime‚1729.922438458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167453800028160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77105727 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33045000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163098052597248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3055635940864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3144587131392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11144622 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168328247482896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170509418929664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163097843092992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165282060643872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44060729 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168328144294950 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167453782988800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30282999 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77105727 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44060729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3055635940864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170509523998208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30282737 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77115886 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33044998 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165282077683232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784498656011131e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159953280865280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19138374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33044998 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3046169799664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3055723970048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44070886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170509523998208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚351 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167453782988800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167453800028160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3144771731968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3046083651078 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚4349 _runtime‚1729.922438458 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167453800028160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77105727 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33045000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163098052597248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3055635940864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3144587131392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11144622 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168328247482896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170509418929664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163097843092992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165282060643872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44060729 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168328144294950 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167453782988800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30282999 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77105727 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44060729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3055635940864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170509523998208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30282737 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77115886 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33044998 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165282077683232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784498656011131e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159953280865280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19138374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33044998 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3046169799664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3055723970048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172373630556160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30613758 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚354 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161702343570432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33403729 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33403729 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164881359756800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170168132326988 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169284752022528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44547583 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11265174 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMof­ ·nitor.on_train_batch_start/active.large_pool.freed‚33403727 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3088895572992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77961455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164881156536832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3088895572992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3179016186368 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚169284734983168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161702318666752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19348584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚169284734983168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172373630556160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455ùàt‚Ø'ÏÒO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚169284734983168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161702318666752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19348584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚169284734983168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172373630556160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚4399 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77951310 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3178837870080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 $ _timestamp‚1.784498676200197e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3088983595008 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172373630556160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30613758 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚354 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161702343570432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33403729 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33403729 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164881359756800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170168132326988 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169284752022528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44547583 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11265174 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33403727 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3088895572992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77961455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164881156536832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3088895572992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3179016186368 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816â£eÝ2Ð- _wandbruntime‚1750  _runtime‚1750ÿ¦‹RÞ&ÑØM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172411675033088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079913126486 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3089573682688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77978688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7844986764707127e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172411780101632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172411780101632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170205680346304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44567638 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚169322101350400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30620482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3089661711872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19352874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11267608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77978688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77968529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33411048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11267863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161738038727680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3179532796928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167125664031872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3079999275072 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _step‚355 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚169322118389760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44567638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161738013824000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚169322101350400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30620740 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3089573682688 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚02…|eã&ÒÝM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172411675033088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079913126486 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3089573682688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77978688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7844986764707127e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172411780101632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172411780101632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170205680346304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44567638 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚169322101350400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30620482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3089661711872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19352874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11267608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77978688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77968529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33411048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11267863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161738038727680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3179532796928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167125664031872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3079999275072 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _step‚355 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚169322118389760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44567638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161738013824000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚169322101350400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30620740 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3089573682688ã×oü—2Ó- _wandbruntime‚1750  _runtime‚1750%–èŠÔ€ epoch‚5 /loss/FRt1/detection_loss‚0.6360696233221917 trainer/global_step‚4399 $ _timestamp‚1.784498676472386e+09 _step‚356 0loss/FRt1/assignment_loss‚2.8337861941511937 0loss/FRt2/assignment_loss‚1.0960784015410088 _runtime‚1750.383115647 1¡ÉMòëloss/FBt2/assignment_loss‚0.09743759317753041 .loss/FRt2/detection_loss‚1.379744715502431 1loss/SRqqt2/detection_loss‚0.6098786401767832 &loss/total_loss‚11.292720693957621 2loss/SRqqt1/assignment_loss‚0.7162376757302921 2loss/SRqqt2/assignment_loss‚0.7270342538808929 1loss/FBt1/assignment_loss‚0.11112957996387897 1loss/SRqqt1/detection_loss‚1.3006690263824716 /loss/FBt1/detection_loss‚0.8520466355098919 /loss/FBt2/detection_loss‚0.93260835461905518ä®( Õ… 1loss/FBt2/assignment_loss‚0.09743759317753041 .loss/FRt2/detection_loss‚1.379744715502431 1loss/SRqqt2/detection_loss‚0.6098786401767832 &loss/total_loss‚11.292720693957621 2loss/SRqqt1/assignment_loss‚0.7162376757302921 2loss/SRqqt2/assignment_loss‚0.7270342538808929 1loss/FBt1/assignment_loss‚0.11112957996387897 1loss/SRqqt1/detection_loss‚1.3006690263824716 /loss/FBt1/detection_loss‚0.8520466355098919 /loss/FBt2/detection_loss‚0.9326083546190551 epoch‚5 /loss/FRt1/detection_loss‚0.6360696233221917 trainer/global_step‚4399 $ _timestamp‚1.784498676472386e+09 _step‚356 0loss/FRt1/assignment_loss‚2.8337861941511937 0loss/FRt2/assignment_loss‚1.0960784015410088 _runtime‚1750.383115647ä á¸PÖ‚(:Å ù“õÒÑŸ€, network.recv‚544424% disk./.usageGB‚1166.4541549682617 disk.nvme1n1.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚1203.32568359375 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0.015625 disk.dm-0.out‚13859.39453125( proc.memory.availableMB‚ 130.74609375" proc.memory.rssMB‚ 2831.6484375) disk./.usagePercent‚75.97812440763796 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.015625$ disk.nvme0n1.out‚10028.478515625' disk.nvme1c1n1.out‚5142.58935546875% disk.nvme1n1.out‚5142.58935546875$ memory_percent‚99.20198917388916* proc.memory.percent‚17.283010482788086 network.sent‚ 5.811949e+06 disk.rbd0.in‚0& disk.nvme0c0n1.out‚10028.478515625 disk.nvme0n1.in‚0.015625 disk.md0.out‚ 50.703125 cpu‚26.309705835767854 proc.cpu.threads‚17Ë;»Ñׂ(:„ ù“õÒØð¬, gpu.0.gpu‚49.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚137.589' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚45.863 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñNµTØ‚(:É ˆ”õÒÎœÍ+ disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚5143.38037109375 disk.md0.out‚ 51.15234375" disk.dm-1.out‚1203.44287109375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚10033.1865234375 network.recv‚548768' disk.nvme1c1n1.out‚5143.38037109375" disk.dm-0.out‚13863.8115234375 disk.dm-1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.283010482788086) disk./.usagePercent‚75.97812788627411 disk.nvme0n1.in‚0.015625 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚99.20074939727783" proc.memory.rssMB‚ 2831.6484375 proc.cpu.threads‚17% disk./.usageGB‚1166.4542083740234 disk.dm-0.in‚0.015625 disk.nvme0c0n1.in‚0.015625( proc.memory.availableMB‚ 130.94921875 cpu‚26.306271610116763 network.sent‚ 5.8605e+06% disk.nvme0n1.out‚10033.18652343757‹·³›Ù‚(: ˆ”õÒ’†‹, gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.543' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.181000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%N±o2Ú- _wandbruntime‚1770  _runtime‚1770Q×!6ÇÛÁ _step‚357 & lr-AdamW/pg1‚9.960380128679667e-05 & lr-AdamW/pg2‚9.960380128679667e-05 trainer/global_step‚4449 % _timestamp‚1.7844986961933367e+09 _runtime‚1770.104299075Ê¥#²ÌÜÆ & lr-AdamW/pg2‚9.960380128679667e-05 trainer/global_step‚4449 % _timestamp‚1.7844986961933367e+09 _runtime‚1770.104299075 _step‚357 & lr-AdamW/pg1‚9.960380128679667e-05åÞÛu2Ý- _wandbruntime‚1770  _runtime‚1770L-4Ô'ÞÎO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844986961943636e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213991031296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚1770.105382077 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172045462002562 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45044335 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33769779 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163487101432832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45044335 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213804326400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168932412391472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19563087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174275991720960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11388058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33769779 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171153053344768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78824257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19563084 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166701092464128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174275886659584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3122921336832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45054478 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168932395352112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33769777 trainer/global_step‚4449 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78814112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172045565190396 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3113152798924 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30950906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45054478 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3122921336832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171153053344768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163487076529152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0B Ù'ßÓO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3122833314816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168932395352112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33769777 trainer/global_step‚4449 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78814112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172045565190396 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3113152798924 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30950906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45054478 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3122921336832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171153053344768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163487076529152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844986961943636e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213991031296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚1770.105382077 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172045462002562 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45044335 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33769779 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163487101432832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45044335 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213804326400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168932412391472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19563087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174275991720960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11388058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33769779 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171153053344768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78824257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19563084 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166701092464128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174275886659584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3122921336832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45054478 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚1770.375630272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78831331 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11390511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214677537792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3123599453696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78831331 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33777098 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171190419712000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33777098 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3113828750480 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78841490 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163522771686400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166737474127872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3123511424512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171190436751360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172083010021766 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45064390 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45054233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33777100 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30957648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚171190419712000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33777100 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78841490 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19567377 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171190436751360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0j‘%êã&âÝM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172083113209712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚4449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174314036205056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11390274 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166737281400832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30957648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚171190419712000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33777100 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78841490 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19567377 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171190436751360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3113742601894 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚1770.375630272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78831331 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11390511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214677537792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3123599453696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78831331 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33777098 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171190419712000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33777098 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3113828750480 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78841490 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163522771686400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166737474127872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3123511424512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171190436751360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172083010021766 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45064390 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45054233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33777100 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3146982274332 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173021388745728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3248782241280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176178142763008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31289154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168520616632832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170776015779472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45551230 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19777584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _runtime‚1789.471330889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176178247824384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45551230 trainer/global_step‚4499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45541087 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11511333 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34135827 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173021388745728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170775998740112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176178247824384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173021371706368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168520813561344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173922894865970 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173021371706368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156771056640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34135827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11511567 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7844987155604606e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176178142763008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3156859078656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19777587 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173021388745728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170775998740112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176178247824384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173021371706368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168520813561344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173922894865970 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173021371706368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156771056640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34135827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11511567 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7844987155604606e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176178142763008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3156859078656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19777587 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3146982274332 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173021388745728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3248782241280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176178142763008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31289154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168520616632832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170776015779472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45551230 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19777584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _runtime‚1789.471330889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0êR»¿2î- _wandbruntime‚1790  _runtime‚1790Q"2•Þ&ïØM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79704292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19781877 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173058738073600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3157537195520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170812870807872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11513978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173960546073120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34143148 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19781874 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34143148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45550985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168557006725120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165307529548800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170812887847232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173058755112960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3249659679744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31295610 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3147658225888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176216292308480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173058738073600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34143150åYûZã&ðÝM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173960442885174 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176216187239936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173058738073600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3157537195520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170812870807872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11513978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173960546073120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34143148 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19781874 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34143148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45550985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168557006725120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165307529548800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170812887847232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173058755112960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3249659679744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31295610 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3147658225888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176216292308480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173058738073600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34143150 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79704292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19781877 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3190708798464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11635020 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80539716 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178080398866432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174889707107328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174889690067968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3283727656960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178080503927808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31626858 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170340319910912 _step‚366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3190796820480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3180811749740 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚178080503927808 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3283908070400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46047982 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46037839 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167056617157632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175800430917212 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174889690067968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19992084 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170340525228032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172619619167472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174889707107328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46037839 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46047982ŽÞÙ'ûÓO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80549861 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31627107 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167056592253952 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3283908070400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46047982 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46037839 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167056617157632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175800430917212 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174889690067968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19992084 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170340525228032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172619619167472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174889707107328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46037839 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46047982 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11634774 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 % _timestamp‚1.7844987353040507e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175800327729378 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3180725601266 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172619602128112 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚1809.215295958 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19992087 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3190796820480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34501879 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3190708798464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11635020 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80539716 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178080398866432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174889707107328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174889690067968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3283727656960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178080503927808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31626858 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170340319910912 _step‚366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3190796820480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3180811749740 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚178080503927808 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚178118548411904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80567094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19996377 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170376715446784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174927056435200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31633658 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172656474195872 _step‚367 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34509200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46047737 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175837978936528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3191474937344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178118443343360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3191386908160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167092312314880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚4549 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175837875748582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 $ _timestamp‚1.784498735576433e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34509200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11637284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3181401552710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80567094 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174927073474560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11637531 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3191474937344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31633908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174927073474560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3284610539008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80556935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1809.487548978 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80556935 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19996374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3191386908160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34509198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46057894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167092287411200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455› ª:Û!þÜM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175837875748582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 $ _timestamp‚1.784498735576433e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34509200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11637284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3181401552710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80567094 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174927073474560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11637531 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3191474937344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31633908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174927073474560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3284610539008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80556935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1809.487548978 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80556935 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19996374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3191386908160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34509198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46057894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167092287411200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚178118548411904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34509198 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚178118548411904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80567094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19996377 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170376715446784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174927056435200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31633658 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172656474195872 _step‚367 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34509200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46047737 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175837978936528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3191474937344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178118443343360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XFDeviceStatsMonitor.on_train_batch_end/allocaY³Âted_bytes.small_pool.freed‚ 3191386908160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167092312314880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚4549 FÄ…¾ _runtime‚1829.105347066 _step‚369 % lr-AdamW/pg1‚9.94244463064869e-05 % lr-AdamW/pg2‚9.94244463064869e-05 trainer/global_step‚4599 $ _timestamp‚1.784498755194403e+09• $Ɇà % lr-AdamW/pg1‚9.94244463064869e-05 % lr-AdamW/pg2‚9.94244463064869e-05 trainer/global_step‚4599 $ _timestamp‚1.784498755194403e+09 _runtime‚1829.105347066 _step‚369ñ|ˆm2‡- _wandbruntime‚1829  _runtime‚1829 ØÔÔ'ˆÎO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3318718732800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20206584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168841350116352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174463205516112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34867927 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179982760031232 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176758025468928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31963796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179982760031232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3214641225148 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4599 _step‚370 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179982654969856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11757212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168841375020032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177677760592786 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3224734562304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3224734562304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179982654969856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11757445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172160068849152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46534591 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177677863780620 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81402518 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3214555076674 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46544734 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Ü›ÙIÙ'‰ÓO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3224734562304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179982654969856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11757445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172160068849152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46534591 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177677863780620 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81402518 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3214555076674 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46544734 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3318718732800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20206584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168841350116352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174463205516112 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34867927 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179982760031232 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176758025468928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31963796 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179982760031232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3214641225148 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4599 _step‚370 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179982654969856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11757212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168841375020032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177677760592786 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3224734562304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0òþ|!c2Š- _wandbruntime‚1829  _runtime‚1829rI뜋×M REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34875250 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34875248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81429896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81419737 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172196462531584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚180020804515328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚4599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46544489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225324649984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3215317176704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3225412679168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168877045273600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3319599761408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31970559 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34875248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3225412679168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1829.378121356 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46554646 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225324649984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180020699446784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3215231028118 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3319417257984 ZFDeviceStatsMonitor.on_train_batch_end]ÞãA /requested_bytes.large_pool.freed‚174500077583872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176795374796800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31970793 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81429896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11759685 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172196669938688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177715411799936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46544489 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330Ç.Áâ&ŒÜM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176795374796800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31970793 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81429896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11759685 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172196669938688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚371 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177715411799936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46544489 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34875250 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34875248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81429896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81419737 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172196462531584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚180020804515328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚4599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46544489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225324649984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3215317176704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3225412679168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168877045273600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3319599761408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31970559 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34875248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3225412679168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1829.378121356 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46554646 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225324649984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180020699446784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3215231028118 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3319417257984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174500077583872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482ó½Ky2- _wandbruntime‚1829  _runtime‚1829Ÿx쎀 0loss/FRt2/assignment_loss‚1.2137315627690908 epoch‚5 trainer/global_step‚4599 _step‚372 1loss/FBt1/assignment_loss‚0.12158873113092787 /loss/FRt2/detection_loss‚1.4599044306426665 % _timestamp‚1.7844987554680994e+09 /loss/FRt1/assignment_loss‚2.756336041502079 2loss/SRqqt1/assignment_loss‚0.8144712382193767 .loss/FRt1/detection_loss‚0.557770839115776 1loss/SRqqt2/detection_loss‚0.6780964357677199 /loss/FBt1/detection_loss‚0.5622548442655082 /loss/FBt2/detection_loss‚0.9264516636141978 &loss/total_loss‚11.032032898236128 2loss/SRqqt2/assignment_loss‚0.7028636314238057 1loss/FBt2/assignment_loss‚0.10999277499068841 1loss/SRqqt1/detection_loss‚1.1285707047942912 _runtime‚1829.378819387¾a5 … .loss/FRt1/detection_loss‚0.557770839115776 1loss/SRqqt2/detection_loss‚0.6780964357677199 /loss/FBt1/detection_loss‚0.5622548442655082 /loss/FBt2/detection_loss‚0.9264516636141978 &loss/total_loss‚11.032032898236128 2loss/SRqqt2/assignment_loss‚0.7028636314238057 1loss/FBt2/assignment_loss‚0.10999277499068841 1loss/SRqqt1/detection_loss‚1.1285707047942912 _runtime‚1829.378819387 0loss/FRt2/assignment_loss‚1.2137315627690908 epoch‚5 trainer/global_step‚4599 _step‚372 1loss/FBt1/assignment_loss‚0.12158873113092787 /loss/FRt2/detection_loss‚1.4599044306426665 % _timestamp‚1.7844987554680994e+09 /loss/FRt1/assignment_loss‚2.756336041502079 2loss/SRqqt1/assignment_loss‚0.8144712382193767ô«„ÔûW‚(:Ì Ä”õÒØ×ô+ disk.md0.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10053.55126953125$ memory_percent‚98.91345500946045 proc.cpu.threads‚17% disk./.usageGB‚1166.4587593078613' disk.nvme1c1n1.out‚5147.05322265625# disk.dm-0.out‚13883.89208984375 disk.dm-1.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2779.390625 network.sent‚ 6.06597e+06 network.recv‚565703& disk.nvme0n1.out‚10053.55126953125 disk.md0.out‚ 52.77734375 disk.nvme0c0n1.in‚0.015625* proc.memory.percent‚16.964054107666016 cpu‚26.293307979156726 disk.nvme0n1.in‚0.015625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚5147.05322265625 disk.dm-0.in‚0.015625" disk.dm-1.out‚1203.90966796875( proc.memory.availableMB‚ 178.01953125) disk./.usagePercent‚75.97842431577016 disk.nvme1n1.in‚0ÜZ~š‘‚(: Ä”õÒË€¡, gpu.0.gpu‚83.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚187.36' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.45333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ɾ§©S’‚(:È Ó”õÒóœÓ+ disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚5147.54736328125 disk.md0.in‚0 disk.nvme0c0n1.in‚0.015625( disk.nvme0c0n1.out‚10054.37646484375 proc.memory.rssMB‚ 2779.40625% disk./.usageGB‚1166.2202072143555 disk.md0.out‚53.09375( proc.memory.availableMB‚ 178.12890625 network.sent‚ 6.11623e+06 disk.dm-0.in‚0.015625 disk.dm-1.in‚0 disk.nvme0n1.in‚0.015625& disk.nvme0n1.out‚10054.37646484375 cpu‚26.295657002748825 proc.cpu.threads‚17 network.recv‚570326' disk.nvme1c1n1.out‚5147.54736328125# disk.dm-0.out‚13884.40771484375" disk.dm-1.out‚1204.03173828125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.91278743743896* proc.memory.percent‚16.964149475097656) disk./.usagePercent‚75.96288599344442,—æS›“‚(: Ó”õÒô«€, gpu.0.gpu‚58.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚165.016' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚55.005333333333326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åŠL}2”- _wandbruntime‚1848  _runtime‚1848(øÈÇ•Á _runtime‚1848.876701297 _step‚373 & lr-AdamW/pg1‚9.936310490692671e-05 & lr-AdamW/pg2‚9.936310490692671e-05 trainer/global_step‚4649 % _timestamp‚1.7844987749655921e+09‹´Ì–Æ _runtime‚1848.876701297 _step‚373 & lr-AdamW/pg1‚9.936310490692671e-05 & lr-AdamW/pg2‚9.936310490692671e-05 trainer/global_step‚4649 % _timestamp‚1.7844987749655921e+09õ‘ýÕ72—- _wandbruntime‚1848  _runtime‚1848å~kÔ'˜ÎO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3258672304128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3258584282112 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32301649 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178626326791168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258584282112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181885016134656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20421087 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3353872085504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3258672304128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179555193456194 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82265320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11880315 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181884911073280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32301399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181884911073280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35233977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170626132882432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3248384552082 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47031343 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20421084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11880562 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3353689574912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7844987749678035e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3258672304128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3258584282112 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32301649 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178626326791168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258584282112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181885016134656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_´ 0bRpool.allocated‚20421087 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3353872085504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3258672304128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179555193456194 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82265320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11880315 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181884911073280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32301399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181884911073280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35233977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170626132882432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3248384552082 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47031343 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20421084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11880562 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3353689574912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7844987749678035e+09 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47041241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20425377 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179592844663344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176343680971872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178663710197760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47051398 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170661828039680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3249146652112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3259350420992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3249060503526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178663693158400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3259350420992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35241298 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179592741475398 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82292698 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259262391808 trainer/global_step‚4649 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82282539 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181923060618752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259262391808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174016188731392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47041241 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82282539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3354385595392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47051398 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35241300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176343698011232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35241300 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32308334 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11882957 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3354572293120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚1849.141838807 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20425374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784498775230635e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82292698 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181922955550208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32308091Ó>³â&œÜM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259262391808 trainer/global_step‚4649 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82282539 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181923060618752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259262391808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174016188731392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47041241 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82282539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3354385595392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47051398 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35241300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176343698011232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35241300 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32308334 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11882957 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3354572293120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚1849.141838807 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20425374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784498775230635e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82292698 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181922955550208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32308091 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47041241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20425377 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179592844663344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176343680971872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178663710197760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47051398 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170661828039680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3249146652112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3259350420992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3249060503526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178663693158400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3259350420992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35241298 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179592741475398 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82292698 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0÷8ÓA2- _wandbruntime‚1849  _runtime‚1849T_Ú`ž /loss/FRt1/detection_loss‚0.6115656423850738 /loss/FRt2/detection_loss‚1.5062019581941641 /loss/FBt1/detection_loss‚0.8286690503358841 % _timestamp‚1.7844987752316341e+09 _runtime‚1849.142241448 _step‚376 /loss/FRt2/assignment_loss‚1.164316712685351 1loss/SRqqt1/detection_loss‚1.1567683299582014 /loss/FBt2/detection_loss‚0.6917169627425719 &loss/total_loss‚10.770202846427674 epoch‚5 2loss/SRqqt1/assignment_loss‚0.6305594062760798 1loss/FBt1/assignment_loss‚0.08136372005009776 1loss/SRqqt2/detection_loss‚0.6384445082978046 1loss/FBt2/assignment_loss‚0.06583564958180556 trainer/global_step‚4649 0loss/FRt1/assignment_loss‚2.7293280657757535 2loss/SRqqt2/assignment_loss‚0.6654328401448866Cé« Ÿ† /loss/FBt1/detection_loss‚0.8286690503358841 % _timestamp‚1.7844987752316341e+09 _runtime‚1849.142241448 _step‚376 /loss/FRt2/assignment_loss‚1.164316712685351 1loss/SRqqt1/detection_loss‚1.1567683299582014 /loss/FBt2/detection_loss‚0.6917169627425719 &loss/total_loss‚10.770202846427674 epoch‚5 2loss/SRqqt1/assignment_loss‚0.6305594062760798 1loss/FBt1/assignment_loss‚0.08136372005009776 1loss/SRqqt2/detection_loss‚0.6384445082978046 1loss/FBt2/assignment_loss‚0.06583564958180556 trainer/global_step‚4649 0loss/FRt1/assignment_loss‚2.7293280657757535 2loss/SRqqt2/assignment_loss‚0.6654328401448866 /loss/FRt1/detection_loss‚0.6115656423850738 /loss/FRt2/detection_loss‚1.5062019581941641øö@à.^ ‚(:Ó â”õÒݪº+" disk.dm-1.out‚1204.14501953125$ memory_percent‚98.91307353973389( proc.memory.availableMB‚ 178.08203125" proc.memory.rssMB‚ 2779.4140625 cpu‚26.298900790260035 proc.cpu.threads‚17) disk./.usagePercent‚75.96273864835544 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10060.26708984375 disk.md0.out‚ 53.32421875# disk.dm-0.out‚13899.49365234375 disk.dm-1.in‚0' disk.nvme1c1n1.out‚5157.36376953125 disk.nvme1n1.in‚0 disk.md0.in‚0* proc.memory.percent‚16.964197158813477 network.sent‚ 6.119152e+06 network.recv‚573631$ disk./.usageGB‚1166.217945098877 disk.nvme0c0n1.in‚ 0.01953125 disk.nvme0n1.in‚ 0.01953125& disk.nvme0n1.out‚10060.26708984375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚5157.36376953125 disk.rbd0.out‚0 disk.dm-0.in‚ 0.01953125¤¨Ÿ›¡‚(: â”õÒÁ½‘, gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.544' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.181333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04e=L2¢- _wandbruntime‚1868  _runtime‚1868K$QðÇ£Á & lr-AdamW/pg1‚9.930098693143534e-05 & lr-AdamW/pg2‚9.930098693143534e-05 trainer/global_step‚4699 % _timestamp‚1.7844987944585898e+09 _runtime‚1868.369522781 _step‚377›wcÌ¤Æ _runtime‚1868.369522781 _step‚377 & lr-AdamW/pg1‚9.930098693143534e-05 & lr-AdamW/pg2‚9.930098693143534e-05 trainer/global_step‚4699 % _timestamp‚1.7844987944585898e+09ùϤWV2¥- _wandbruntime‚1868  _runtime‚1868#GºN&¦ÎO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181432626319602 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47528095 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20635584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3292610045952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12003497 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3388673286656 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83128122 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚378 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183787272238080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181432729507436 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178150429331472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388851602944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 _runtime‚1868.370469463 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20635587 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172410890744832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32639084 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12003265 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47528095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3292522023936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3292610045952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.cu§¼ ·®rrent‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32638849 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83138267 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚178150412292112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183787167176704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180494645152768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175799539127808 % _timestamp‚1.7844987944595523e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181432729507436 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178150429331472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388851602944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 _runtime‚1868.370469463 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20635587 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172410890744832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32639084 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12003265 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47528095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3292522023936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3292610045952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32638849 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83138267 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚178150412292112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183787167176704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180494645152768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175799539127808 % _timestamp‚1.7844987944595523e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚1868.628608684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175836138728448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3282976127520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83155500 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3282889978934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚379 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32645520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181470174338806 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35607348 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47537993 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178187284359872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47537993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20639874 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178187301399232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180532011520000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83155500 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35607350 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20639877 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32645760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3293200133632£éDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83155500 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3282889978934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚379 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32645520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181470174338806 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35607348 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47537993 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178187284359872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47537993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20639874 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178187301399232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180532011520000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83155500 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35607350 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20639877 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32645760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3293200133632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172446585902080 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180532011520000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35607350 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚1868.628608684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175836138728448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3282976127520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816û9'‚b2«- _wandbruntime‚1868  _runtime‚1868ˆ†ä;¬€ 1loss/FBt1/assignment_loss‚0.07415863913719736 2loss/FBt2/assignment_loss‚0.046671820412439044 /loss/FRt1/detection_loss‚0.6801752016428776 1loss/SRqqt2/detection_loss‚0.6705871230163515 /loss/FBt1/detection_loss‚1.6414453415421857 $ _timestamp‚1.784498794718473e+09 1loss/SRqqt1/assignment_loss‚0.758465543421196 &loss/total_loss‚11.785194857188532 /loss/FRt2/detection_loss‚1.4156393082908112 1loss/SRqqt1/detection_loss‚1.1606162152740926 _runtime‚1868.629095755 /loss/FRt1/assignment_loss‚2.741557338232294 2loss/SRqqt2/assignment_loss‚0.7272307916829656 epoch‚5 trainer/global_step‚4699 _step‚380 /loss/FBt2/detection_loss‚0.6979277529773147 0loss/FRt2/assignment_loss‚1.1707197815588075»ÛR ­… /loss/FBt2/detection_loss‚0.6979277529773147 0loss/FRt2/assignment_loss‚1.1707197815588075 1loss/FBt1/assignment_loss‚0.07415863913719736 2loss/FBt2/assignment_loss‚0.046671820412439044 /loss/FRt1/detection_loss‚0.6801752016428776 1loss/SRqqt2/detection_loss‚0.6705871230163515 /loss/FBt1/detection_loss‚1.6414453415421857 $ _timestamp‚1.784498794718473e+09 1loss/SRqqt1/assignment_loss‚0.758465543421196 &loss/total_loss‚11.785194857188532 /loss/FRt2/detection_loss‚1.4156393082908112 1loss/SRqqt1/detection_loss‚1.1606162152740926 _runtime‚1868.629095755 /loss/FRt1/assignment_loss‚2.741557338232294 2loss/SRqqt2/assignment_loss‚0.7272307916829656 epoch‚5 trainer/global_step‚4699 _step‚380üSž+R®‚(:Ç ñ”õÒûÖ³+ disk.md0.out‚ 53.77734375$ memory_percent‚98.91343116760254" proc.memory.rssMB‚ 2779.4140625% disk.nvme0c0n1.out‚10061.63671875 network.recv‚578116 disk.dm-0.in‚ 0.01953125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.01953125 disk.nvme1c1n1.in‚0 disk.md0.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚13900.4150390625' proc.memory.availableMB‚ 178.0234375* proc.memory.percent‚16.964197158813477 cpu‚26.305399020330935 disk.nvme0n1.in‚ 0.01953125# disk.nvme0n1.out‚10061.63671875 network.sent‚ 6.182476e+06) disk./.usagePercent‚75.96272970329102 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme1c1n1.out‚5158.138671875 disk.nvme1n1.in‚0% disk./.usageGB‚1166.2178077697754! disk.dm-1.out‚1204.3935546875# disk.nvme1n1.out‚5158.138671875<æT Ž¯‚(:ƒ ñ”õÒþâ‰, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.393' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.131 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ­/°2°- _wandbruntime‚1887  _runtime‚1887ÖDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3316043502898 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185689528341504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48024847 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35966077 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20850087 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3326547787776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84001069 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12126002 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182362963514368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35966079 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179994015680112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185689528341504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177619247269376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183310162370844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35966077 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32976086 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179994032719472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844988135554001e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182362963514368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185689423280128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚4749 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182362980553728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35966079 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48034990 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚382 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185689423280128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3326459765760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174195623703552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182362980553728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20850084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12126239 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3423623565824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183310059183010 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3423799784960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3326547787776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174195648607232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3316129651372 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20850087 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3326547787776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84001069 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12126002 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182362963514368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35966079 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179994015680112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185689528341504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177619247269376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183310162370844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35966077 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32976086 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179994032719472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844988135554001e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182362963514368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185689423280128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚4749 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182362980553728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35966079 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48034990 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚382 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185689423280128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3326459765760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174195623703552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182362980553728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20850084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12126239 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3423623565824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183310059183010 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3423799784960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3326547787776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174195648607232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3316129651372 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3316043502898 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104þHC%2¶- _wandbruntime‚1887  _runtime‚1887ø`P ·ØM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185727572825600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177655632176128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183347710390160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84008143 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48034745 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84018302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84018302 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35973400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185727572825600 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182400329881600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177655843777536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 trainer/global_step‚4749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3327225904640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185727467757056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32982952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183347607202214 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48034745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3424313315328 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20854374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84008143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12128338 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48044902 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844988138120883e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174231343764480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3327225904640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48044902 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32982712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180030887747872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3316719454342 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182400329881600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3327137875456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3316805602928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚1887.723178673 _step‚383 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3327137875456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20854377 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12128575 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180030904787232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35973400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208­·<ã&¸ÝM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84008143 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12128338 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48044902 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844988138120883e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174231343764480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3327225904640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48044902 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32982712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180030887747872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3316719454342 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182400329881600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3327137875456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3316805602928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚1887.723178673 _step‚383 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3327137875456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20854377 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12128575 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180030904787232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35973400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185727572825600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177655632176128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183347710390160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84008143 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48034745 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84018302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84018302 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35973400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185727572825600 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182400329881600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177655843777536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 trainer/global_step‚4749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3327225904640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185727467757056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32982952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183347607202214 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48034745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3424313315328 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20854374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÿ­ïž2¹- _wandbruntime‚1887  _runtime‚1887Úôùº /loss/FRt2/assignment_loss‚1.175448789512902 1loss/FBt2/assignment_loss‚0.04312192151481516 1loss/SRqqt1/detection_loss‚1.1908369689523208 &loss/total_loss‚11.201805756281175 trainer/global_step‚4749 /loss/FRt1/assignment_loss‚2.823576396047059 2loss/SRqqt1/assignment_loss‚0.6525054576826824 0loss/FRt1/detection_loss‚0.49684360318719034 /loss/FRt2/detection_loss‚1.3495546593412495 epoch‚5 2loss/SRqqt2/assignment_loss‚0.6998207614839054 1loss/FBt1/assignment_loss‚0.16849080376074277 1loss/SRqqt2/detection_loss‚0.7223977521314974 /loss/FBt1/detection_loss‚0.9835300163527869 % _timestamp‚1.7844988138130352e+09 /loss/FBt2/detection_loss‚0.8956786263140236 _runtime‚1887.724121635 _step‚384_²Â4 »† /loss/FRt1/assignment_loss‚2.823576396047059 2loss/SRqqt1/assignment_loss‚0.6525054576826824 0loss/FRt1/detection_loss‚0.49684360318719034 /loss/FRt2/detection_loss‚1.3495546593412495 epoch‚5 2loss/SRqqt2/assignment_loss‚0.6998207614839054 1loss/FBt1/assignment_loss‚0.16849080376074277 1loss/SRqqt2/detection_loss‚0.7223977521314974 /loss/FBt1/detection_loss‚0.9835300163527869 % _timestamp‚1.7844988138130352e+09 /loss/FBt2/detection_loss‚0.8956786263140236 _runtime‚1887.724121635 _step‚384 /loss/FRt2/assignment_loss‚1.175448789512902 1loss/FBt2/assignment_loss‚0.04312192151481516 1loss/SRqqt1/detection_loss‚1.1908369689523208 &loss/total_loss‚11.201805756281175 trainer/global_step‚4749€ž¿¨:Q¼‚(:Æ €•õÒ•Ý÷+ cpu‚26.313091570249306 proc.cpu.threads‚17' disk.nvme1c1n1.out‚5215.57470703125% disk./.usageGB‚1166.5200805664062 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚5215.57470703125 disk.md0.in‚0& proc.memory.availableMB‚ 177.984375 disk.nvme0c0n1.in‚ 0.01953125 disk.nvme0n1.in‚ 0.01953125# disk.dm-0.out‚13964.88916015625$ memory_percent‚98.91366958618164* proc.memory.percent‚16.964197158813477" disk.nvme0n1.out‚ 10069.3671875 network.sent‚ 6.257783e+06 network.recv‚582505! disk.dm-1.out‚1204.5185546875$ disk.nvme0c0n1.out‚ 10069.3671875 disk.nvme1n1.in‚0 disk.md0.out‚54.03125 disk.dm-0.in‚ 0.01953125) disk./.usagePercent‚75.98241853550928 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2779.4140625iÊ{ù™½‚(:Ž €•õÒ¢ï , gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.641' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.21366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¯b³R¾‚(:Ç •õÒ´íã+" disk.dm-1.out‚1204.72607421875 disk.rbd0.out‚0% disk.nvme0n1.out‚10228.5595703125 disk.nvme1c1n1.in‚0 disk.md0.out‚ 54.546875 disk.dm-0.in‚ 0.01953125 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme1c1n1.out‚5250.388671875# disk.dm-0.out‚14157.58447265625' proc.memory.availableMB‚ 177.9765625* proc.memory.percent‚16.964244842529297 cpu‚26.31664307684105 network.recv‚587084 disk.nvme0c0n1.in‚ 0.01953125# disk.nvme1n1.out‚5250.388671875$ memory_percent‚98.91371726989746 proc.cpu.threads‚17$ disk./.usageGB‚1166.460536956787' disk.nvme0c0n1.out‚10228.5595703125 disk.nvme0n1.in‚ 0.01953125 disk.nvme1n1.in‚0 disk.md0.in‚0! proc.memory.rssMB‚ 2779.421875 network.sent‚ 6.30793e+06) disk./.usagePercent‚75.97854010465964Ÿ.>d›¿‚(: •õÒÇÝ‹, gpu.0.gpu‚69.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚162.218' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚54.072666666666656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02S$2À- _wandbruntime‚1906  _runtime‚1906€9yñÇÁÁ _step‚385 & lr-AdamW/pg1‚9.917442524486345e-05 & lr-AdamW/pg2‚9.917442524486345e-05 trainer/global_step‚4799 % _timestamp‚1.7844988330502732e+09 _runtime‚1906.961314774ÓÝÛlÌÂÆ & lr-AdamW/pg2‚9.917442524486345e-05 trainer/global_step‚4799 % _timestamp‚1.7844988330502732e+09 _runtime‚1906.961314774 _step‚385 & lr-AdamW/pg1‚9.917442524486345e-05|EÊn2Ã- _wandbruntime‚1906  _runtime‚19065²µÔ'ÄÎO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184231298915328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48531742 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175980381565952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3360485529600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3349959126780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚4799 % _timestamp‚1.7844988330510592e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21064584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36332129 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33313938 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179438973066752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3458776108544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181837636107472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33313697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187591679383552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184231298915328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3360485529600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187591679383552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12249351 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48531742 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36332129 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184231281875968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187591784444928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3360397507584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48521599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84853726 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185187492046418 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48521599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187591784444928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179439182578176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175980406469632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181837619068112 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3349872978306 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184231281875968 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36332127 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3360397507584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36332127 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3360397507584 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3360485529600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3349959126780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚4799 % _timestamp‚1.7844988330510592e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21064584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36332129 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33313938 ZLDeviceStatsMonitor.on_train_batch_start/inactiv„Ì€'àe_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179438973066752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3458776108544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181837636107472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33313697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187591679383552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184231298915328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3360485529600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187591679383552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12249351 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48531742 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36332129 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184231281875968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187591784444928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3360397507584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48521599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84853726 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185187492046418 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48521599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187591784444928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179439182578176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175980406469632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181837619068112‚0×£2Æ- _wandbruntime‚1907  _runtime‚1907.œÝâÞ&ÇØM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184268665282560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚387 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187629828929024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84881104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184268665282560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36339448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187629828929024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181874491135872 trainer/global_step‚4799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36339448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176016076723200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3350635078336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84881104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36339450 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3361075617280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185225040065622 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3361075617280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179475576177664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3350548929750 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84870945 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36339450 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12251739 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184268648243200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3459474550784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844988333109982e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184268648243200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21068874 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181874508175232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187629723860480 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33320368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185225143253568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3361163646464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84870945 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179475370867712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48531497 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12251494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176016101626880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844988333109982e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184268648243200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21068874 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181874508175232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187629723860480 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33320368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185225143253568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3361163646464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84870945 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179475370867712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48531497 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12251494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176016101626880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84881104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184268665282560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36339448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187629828929024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181874491135872 trainer/global_step‚4799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36339448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176016076723200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3350635078336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84881104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36339450 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3361075617280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185225040065622 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3361075617280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179475576177664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3350548929750 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84870945 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36339450 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12251739 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184268648243200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3459474550784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2ƒÕ{Œ2É- _wandbruntime‚1907  _runtime‚1907絨™Êÿ epoch‚5 0loss/FRt2/assignment_loss‚1.1464464863141377 /loss/FRt1/detection_loss‚0.5695297034406606 /loss/FRt2/detection_loss‚1.3959143645549352 _step‚388 /loss/FRt1/assignment_loss‚2.887582335962812 2loss/SRqqt2/assignment_loss‚0.7057710616201297 1loss/FBt1/assignment_loss‚0.19484092010136414 1loss/SRqqt1/detection_loss‚1.3429173549105013 /loss/FBt2/detection_loss‚0.7957046952180784 trainer/global_step‚4799 % _timestamp‚1.7844988333119376e+09 _runtime‚ 1907.22255629 1loss/FBt2/assignment_loss‚0.00709582466242864 1loss/SRqqt2/detection_loss‚0.6579308078716445 /loss/FBt1/detection_loss‚1.2309886595562682 1loss/SRqqt1/assignment_loss‚0.897279033607005 &loss/total_loss‚11.832001247819964?Ú¿‹ Ë„ &loss/total_loss‚11.832001247819964 epoch‚5 0loss/FRt2/assignment_loss‚1.1464464863141377 /loss/FRt1/detection_loss‚0.5695297034406606 /loss/FRt2/detection_loss‚1.3959143645549352 _step‚388 /loss/FRt1/assignment_loss‚2.887582335962812 2loss/SRqqt2/assignment_loss‚0.7057710616201297 1loss/FBt1/assignment_loss‚0.19484092010136414 1loss/SRqqt1/detection_loss‚1.3429173549105013 /loss/FBt2/detection_loss‚0.7957046952180784 trainer/global_step‚4799 % _timestamp‚1.7844988333119376e+09 _runtime‚ 1907.22255629 1loss/FBt2/assignment_loss‚0.00709582466242864 1loss/SRqqt2/detection_loss‚0.6579308078716445 /loss/FBt1/detection_loss‚1.2309886595562682 1loss/SRqqt1/assignment_loss‚0.897279033607005„FtÝY[Ì‚(:Ð ž•õÒ²‰á+ network.sent‚ 6.310995e+06' disk.nvme1c1n1.out‚5251.58544921875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚5251.58544921875# disk.dm-0.out‚14162.96826171875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚10233.46044921875& disk.nvme0n1.out‚10233.46044921875 disk.md0.in‚0 disk.dm-0.in‚ 0.01953125" disk.dm-1.out‚1204.81201171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.01953125( proc.memory.availableMB‚ 136.62890625" proc.memory.rssMB‚ 2820.6484375$ disk./.usageGB‚1163.743522644043 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚ 0.01953125* proc.memory.percent‚17.215871810913086 proc.cpu.threads‚17 network.recv‚590527( disk./.usagePercent‚75.8015647382539 disk.md0.out‚ 54.8359375$ memory_percent‚99.16608333587646 cpu‚26.339663890271351–#šÍ‚(: ž•õÒƒðº, gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.641' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.21366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÎÔð2Î- _wandbruntime‚1925  _runtime‚1925Äý…x Ïçz 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05990961566567421 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01870577782392502 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980607628822327 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4670000672340393 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024931490421295166 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.816022515296936 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6648187041282654 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02871931530535221 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4745945334434509 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7200000286102295 'particle/f_score‚0.6413372159004211 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6711367964744568 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.724934458732605 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03589896485209465 +jet/accuracy_3_of_4‚0.08289898186922073 6validation_average_jet_accuracy‚0.3412104845046997 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9535688757896423 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.36876553297042847 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 .particle/accuracy_3_of_3‚0.999514102935791 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9896344542503357 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1437455266714096 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05981345474720001 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4842539429664612 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.45263272523880005 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5010378956794739 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997687041759491 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5801454186439514 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959183931350708 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983720183372498 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6669324040412903 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6272606253623962 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5149999856948853 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.759864330291748 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.210216"3ù)õ/7308330536 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.988200843334198 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1184592917561531 % _timestamp‚1.7844988520377328e+09 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6128195524215698 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 4 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1346445530653 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.984603762626648 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14413125813007355 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9896264672279358 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13876503705978394 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 trainer/global_step‚4805 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9117839336395264 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10713471472263336 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3676982522010803 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13125288486480713 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18550299108028412 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13055965304374695 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6180990934371948 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.13042819499969482 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.36742421984672546 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7491353154182434 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.816022515296936 ,jet/accuracy_3_of_3‚0.058839764446020126 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8724590539932251 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5406650900840759 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05985627323389053 +particle/sensitivity‚0.6356045603752136 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6015567779541016 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7771790623664856 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012346841394901276 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.29334864020347595 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13712908327579498 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8573485612869263 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚11.427665730943687 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21840256452560425 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971791505813599 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05220659449696541 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6663222312927246 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9855745434761047 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2222222089767456 _step‚389 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.42017945647239685 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8021367192268372 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.37899747490882874 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23324444890022278 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8875175714492798 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49477773904800415 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7084258794784546 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7350426912307739 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5292156934738159 *jet/accuracy_1_of_2‚0.7752072215080261 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6750004291534424 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1509988158941269 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5972809791564941 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36547428369522095 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9933069348335266 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.952629566192627 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19577723741531372 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05220659449696541 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9972641468048096 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 3"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚ 0.7138671875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5173611044883728 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6443457007408142 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4973958432674408 +particle/specificity‚0.9052697420120239 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5908219218254089 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6216948628425598 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7808167338371277 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16637413203716278 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11491616815328598 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1446603536605835 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1180277094244957 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7298083305358887 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6762911081314087 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5240801572799683 /particle/accuracy_2_of_2‚0.9998950958251953 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5902996063232422 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_1‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8137726783752441 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.059609003365039825 particle/accuracy_2_of_4‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7331940531730652 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8162279725074768 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09739583730697632 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5824910402297974 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.32716047763824463 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.955892026424408 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.036629997193813324 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 .particle/accuracy_4_of_4‚0.956404983997345 _runtime‚ 1925.94909413 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4299705922603607 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9655482769012451 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7887972593307495 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2705692648887634 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6272606253623962 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1381864845752716 *jet/accuracy_1_of_3‚0.9241868257522583 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7031220197677612 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8283464312553406 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15181732177734375 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1525399386882782 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7200000286102295 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1269254833459854 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1521786004304886 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7913344502449036 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997687041759491 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.803070068359375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9643608331680298 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8080199956893921 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33394286036491394 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 )jet/accuracy_2_of_3‚0.511211097240448 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5361951589584351 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13379400968551636 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6158696413040161 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 &particle/accuracy‚0.84736168384552 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9813149571418762 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.739338219165802 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16942279040813446 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13876503705978394 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13009202480316162 particle/accuracy_1_of_3‚1Îv±r=Ðìz 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9855745434761047 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2222222089767456 _step‚389 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.42017945647239685 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8021367192268372 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.37899747490882874 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23324444890022278 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8875175714492798 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49477773904800415 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7084258794784546 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7350426912307739 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5292156934738159 *jet/accuracy_1_of_2‚0.7752072215080261 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6750004291534424 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1509988158941269 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5972809791564941 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36547428369522095 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9933069348335266 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.952629566192627 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19577723741531372 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05220659449696541 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9972641468048096 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 3"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚ 0.7138671875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5173611044883728 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6443457007408142 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4973958432674408 +particle/specificity‚0.9052697420120239 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5908219218254089 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6216948628425598 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7808167338371277 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16637413203716278 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11491616815328598 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1446603536605835 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1180277094244957 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7298083305358887 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6762911081314087 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5240801572799683 /particle/accuracy_2_of_2‚0.9998950958251953 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5902996063232422 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_1‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8137726783752441 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.059609003365039825 particle/accuracy_2_of_4‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7331940531730652 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8162279725074768 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09739583730697632 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5824910402297974 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.32716047763824463 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.955892026424408 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.036629997193813324 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 .particle/accuracy_4_of_4‚0.956404983997345 _runtime‚ 1925.94909413 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4299705922603607 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9655482769012451 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7887972593307495 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2705692648887634 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6272606253623962 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1381864845752716 *jet/accuracy_1_of_3‚0.9241868257522583 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7031220197677612 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8283464312553406 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15181732177734375 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1525399386882782 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7200000286102295 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1269254833459854 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1521786004304886 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7913344502449036 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997687041759491 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.803070068359375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9643608331680298 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8080199956893921 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33394286036491394 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 )jet/accuracy_2_of_3‚0.511211097240448 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5361951589584351 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13379400968551636 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6158696413040161 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 &particle/accuracy‚0.84736168384552 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9813149571418762 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.739338219165802 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16942279040813446 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13876503705978394 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13009202480316162 particle/accuracy_1_of_3‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05990961566567421 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01870577782392502 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980607628822327 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4670000672340393 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024931490421295166 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.816022515296936 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6648187041282654 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02871931530535221 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4745945334434509 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7200000286102295 'particle/f_score‚0.6413372159004211 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6711367964744568 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.724934458732605 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03589896485209465 +jet/accuracy_3_of_4‚0.08289898186922073 6validation_average_jet_accuracy‚0.3412104845046997 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9535688757896423 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.36876553297042847 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 .particle/accuracy_3_of_3‚0.999514102935791 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9896344542503357 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1437455266714096 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05981345474720001 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4842539429664612 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.45263272523880005 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5010378956794739 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997687041759491 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5801454186439514 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959183931350708 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983720183372498 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6669324040412903 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6272606253623962 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5149999856948853 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.759864330291748 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2102167308330536 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.988200843334198 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1184592917561531 % _timestamp‚1.7844988520377328e+09 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6128195524215698 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 4 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1346445530653 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.984603762626648 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14413125813007355 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9896264672279358 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13876503705978394 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 trainer/global_step‚4805 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9117839336395264 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10713471472263336 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3676982522010803 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13125288486480713 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18550299108028412 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13055965304374695 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6180990934371948 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.13042819499969482 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.36742421984672546 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7491353154182434 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.816022515296936 ,jet/accuracy_3_of_3‚0.058839764446020126 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8724590539932251 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5406650900840759 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05985627323389053 +particle/sensitivity‚0.6356045603752136 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6015567779541016 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7771790623664856 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012346841394901276 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.29334864020347595 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13712908327579498 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8573485612869263 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚11.427665730943687 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21840256452560425 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971791505813599 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05220659449696541 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6663222312927246 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1…•¥tÃVÑ‚(:Ë ­•õÒëÎ+( proc.memory.availableMB‚ 194.06640625 network.sent‚ 6.3743e+06) disk./.usagePercent‚75.80150286822497$ disk.nvme0n1.out‚10234.513671875 disk.md0.in‚0# proc.memory.rssMB‚ 2822.67578125* proc.memory.percent‚17.228245735168457% disk./.usageGB‚1163.7425727844238 disk.dm-1.in‚0 disk.dm-1.out‚ 1268.92578125 disk.dm-0.in‚ 0.01953125 cpu‚26.34757567516329 proc.cpu.threads‚17 network.recv‚595282 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5316.26025390625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚5316.26025390625# disk.dm-0.out‚14163.60888671875$ memory_percent‚98.81551265716553 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.01953125& disk.nvme0c0n1.out‚10234.513671875 disk.nvme0n1.in‚ 0.01953125 disk.md0.out‚ 55.29296875t\]šÒ‚(: ­•õÒòˆ§, gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.476' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.15866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þw`É2Ó- _wandbruntime‚1942  _runtime‚1942Ææ²ÍÇÔÁ & lr-AdamW/pg2‚9.910998356092523e-05 trainer/global_step‚4849 % _timestamp‚1.7844988690780325e+09 _runtime‚1942.988968278 _step‚390 & lr-AdamW/pg1‚9.910998356092523e-05WÞðŠÌÕÆ & lr-AdamW/pg1‚9.910998356092523e-05 & lr-AdamW/pg2‚9.910998356092523e-05 trainer/global_step‚4849 % _timestamp‚1.7844988690780325e+09 _runtime‚1942.988968278 _step‚390†bîË2Ö- _wandbruntime‚1942  _runtime‚1942ös ×ÎO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3414755566592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182204848220672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36935647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178690562692096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86121262 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49195760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188053808385744 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3414843588608 L?DeviceStöbgaatsMonitor.on_train_batch_start/active.large_pool.freed‚36935645 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184649711924464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚1942.993367834 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3514073920000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190487124460544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21373164 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3404182609754 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187072263832576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49185617 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚391 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49185617 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚4849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33825013 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3514260624896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21373167 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12452119 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182204636612096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178690587595776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187072280871936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3414843588608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3404096461280 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190487019399168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187072280871936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455EaæÙ'ØÓO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184649711924464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚1942.993367834 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3514073920000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190487124460544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21373164 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3404182609754 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187072263832576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49185617 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚391 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49185617 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚4849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33825013 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3514260624896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21373167 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12452119 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182204636612096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178690587595776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187072280871936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3414843588608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3404096461280 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190487019399168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187072280871936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3414755566592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182204848220672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36935647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178690562692096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86121262 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49195760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188053808385744 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3414843588608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36935645 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0‡WI&¿2Ù- _wandbruntime‚1943  _runtime‚1943U£Þ&ÚØM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184686583992224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49205672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187109630199808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36942966 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188091459592894 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33831851 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182241241521152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844988693370812e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49195515 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182241036211200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3514958768128 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86138481 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33832126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12454397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3415521705472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3514778361856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187109647239168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3415521705472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21377457 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚392 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86148640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36942966 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3404858561310 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1943.248181868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190525063876096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187109647239168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21377454 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49195515 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12454669 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3415433676288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3404772412724 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178726282753024,4ÒÄ“ÛÝM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49205672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187109630199808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36942966 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188091459592894 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33831851 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182241241521152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844988693370812e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49195515 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182241036211200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3514958768128 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86138481 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33832126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12454397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonito…ÀÕèPr.on_train_batch_end/active_bytes.small_pool.allocated‚ 3415521705472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3514778361856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187109647239168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3415521705472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21377457 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚392 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86148640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36942966 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3404858561310 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1943.248181868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190525063876096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187109647239168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21377454 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49195515 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12454669 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3415433676288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3404772412724 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178726282753024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184686583992224ˆËÐ`2Ü- _wandbruntime‚1943  _runtime‚1943<óÞfÝ /loss/FRt1/assignment_loss‚2.663290013337799 2loss/SRqqt1/assignment_loss‚0.7007767032126948 /loss/FRt2/detection_loss‚1.5370344366795246 &loss/total_loss‚11.289447241655587 _step‚393 0loss/FRt2/assignment_loss‚1.1921743408654086 1loss/FBt1/assignment_loss‚0.28290088165640553 1loss/FBt2/assignment_loss‚0.06835472616880207 1loss/SRqqt2/detection_loss‚0.7958352773897968 /loss/FBt1/detection_loss‚1.0082775269945463 /loss/FBt2/detection_loss‚0.6569241224429928 epoch‚6 _runtime‚1943.248674718 2loss/SRqqt2/assignment_loss‚0.6627272979432887 /loss/FRt1/detection_loss‚0.5875944864393171 1loss/SRqqt1/detection_loss‚1.1335574285250105 trainer/global_step‚4849 % _timestamp‚1.7844988693379893e+09 ͬ‚ Þ† 1loss/SRqqt1/detection_loss‚1.1335574285250105 trainer/global_step‚4849 % _timestamp‚1.7844988693379893e+09 /loss/FRt1/assignment_loss‚2.663290013337799 2loss/SRqqt1/assignment_loss‚0.7007767032126948 /loss/FRt2/detection_loss‚1.5370344366795246 &loss/total_loss‚11.289447241655587 _step‚393 0loss/FRt2/assignment_loss‚1.1921743408654086 1loss/FBt1/assignment_loss‚0.28290088165640553 1loss/FBt2/assignment_loss‚0.06835472616880207 1loss/SRqqt2/detection_loss‚0.7958352773897968 /loss/FBt1/detection_loss‚1.0082775269945463 /loss/FBt2/detection_loss‚0.6569241224429928 epoch‚6 _runtime‚1943.248674718 2loss/SRqqt2/assignment_loss‚0.6627272979432887 /loss/FRt1/detection_loss‚0.5875944864393171‰ I6,ß‚(:… ¼•õÒ·ðÖ+ gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚194.094' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚64.698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôPìšZà‚(:Ï ¼•õÒÍ•Î,% disk.nvme1n1.out‚5342.63232421875 disk.md0.out‚ 55.51171875 disk.dm-1.in‚0$ memory_percent‚98.81594181060791 cpu‚26.344637747652637' disk.nvme0c0n1.out‚10243.1201171875% disk.nvme0n1.out‚10243.1201171875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚14173.93701171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.01953125 disk.dm-1.out‚ 1293.09765625( proc.memory.availableMB‚ 193.99609375# proc.memory.rssMB‚ 2822.67578125 network.sent‚ 6.442779e+06) disk./.usagePercent‚75.80150858312724$ disk./.usageGB‚1163.742660522461 disk.nvme0c0n1.in‚ 0.01953125 disk.md0.in‚0 disk.dm-0.in‚ 0.01953125* proc.memory.percent‚17.228245735168457 proc.cpu.threads‚17 network.recv‚599371' disk.nvme1c1n1.out‚5342.63232421875Ð2Bž2á- _wandbruntime‚1961  _runtime‚1961@ EúÆâÀ trainer/global_step‚4899 % _timestamp‚1.7844988875560603e+09 _runtime‚ 1961.46837814 _step‚394 & lr-AdamW/pg1‚9.904476935530796e-05 & lr-AdamW/pg2‚9.904476935530796e-05^¨•´ËãÅ & lr-AdamW/pg1‚9.904476935530796e-05 & lr-AdamW/pg2‚9.904476935530796e-05 trainer/global_step‚4899 % _timestamp‚1.7844988875560603e+09 _runtime‚ 1961.46837814 _step‚394ŠL«éA2ä- _wandbruntime‚1961  _runtime‚1961å²Ó'åÍO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184024393059840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37301695 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192389275502592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192389275502592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37301695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12575460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184024600474112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3448693308416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192389380563968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21587667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188940599233536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12575213 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186493332351824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3448693308416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192389380563968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34163127 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _step‚395 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49682369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86994209 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3437925936688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186493315312464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3448781330432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37301697 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180475320554496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49692512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21587664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189931241249152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188940582194176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86984064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3448781330432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚4899 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188940599233536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188940582194176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250.À¸Ø'æÒO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180475345458176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚189931344436986 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49682369 % _timestamp‚1.7844988875566013e+09 _runtime‚ 1961.46846632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37301697 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚4899 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188940599233536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188940582194176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184024393059840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37301695 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192389275502592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192389275502592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37301695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12575460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184024600474112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3448693308416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192389380563968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21587667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188940599233536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12575213 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186493332351824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3448693308416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192389380563968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34163127 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _step‚395 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49682369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86994209 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3437925936688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186493315312464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3448781330432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37301697 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180475320554496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49692512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21587664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189931241249152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188940582194176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86984064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3448781330432 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37309016 _step‚396 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180511015711744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49702424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 trainer/global_step‚4899 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49702424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12577682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188977965600768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188977965600768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37309018 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1961.723363599 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87011442 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37309016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34169880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3449459447296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21591954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_½##‘pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184060995720192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚192427319979520 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87011442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180511040615424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192427425048064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3449371418112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184060788313088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189968892456302 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188977948561408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12577923 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189968789268356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87001283 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7844988878122933e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3449371418112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3549955104768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3438688036718 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188977948561408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3449459447296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192427425048064 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34169636 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3549772601344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2õkã&éÝM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34169636 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3549772601344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3438601888132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21591957 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87001283 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49692267 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37309018 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚186530204419584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37309016 _step‚396 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180511015711744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49702424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 trainer/global_step‚4899 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49702424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12577682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188977965600768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188977965600768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37309018 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1961.723363599 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87011442 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37309016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34169880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3449459447296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21591954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184060995720192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚192427319979520 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87011442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180511040615424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192427425048064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3449371418112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184060788313088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189968892456302 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188977948561408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12577923 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189968789268356 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87001283 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7844988878122933e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3449371418112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3549955104768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3438688036718 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188977948561408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3449459447296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192427425048064 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194291531606016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190808917595136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3482631050240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21802167 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 _runtime‚1980.288140594 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚194291636667392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3584053153792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37667747 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3471841560570 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21802164 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12698744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87846866 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37667745 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50179121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188336918700464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3482719072256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3482631050240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185844131570688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182260103320576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471755412096 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87846866 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37667745 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50179121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188336918700464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3482719072256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3482631050240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185844131570688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182260103320576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471755412096 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194291531606016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190808917595136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3482631050240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21802167 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 _runtime‚1980.288140594 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚194291636667392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3584053153792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37667747 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3471841560570 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21802164 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12698744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816«*ãÌ2÷- _wandbruntime‚1980  _runtime‚1980ÇW1Ü&øÖM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190846283962368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3483397189120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34507635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182295798477824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188373790768224 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87864085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21806457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12700939 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37675068 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190846266923008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34507393 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472431363540 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚ 1980.54567544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21806454 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚194329681151488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50199176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37675066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483309159936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚194329681151488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191846325319710 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190846283962368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚4949 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194329576082944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191846222131764 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12701178 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3483397189120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 $ _timestamp‚1.784498906634612e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87874244 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190846266923008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185880533139968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188373807807584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483309159936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182295773574144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37675068 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50189019 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3584759565824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87864085 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37675066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50199176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87874244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3472517512126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464PƒÁóá&ùÛM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50199176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87874244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3472517512126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190846283962368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3483397189120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34507635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182295798477824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188373790768224 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87864085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21806457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12700939 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37675068 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190846266923008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34507393 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472431363540 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚ 1980.54567544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21806454 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚194329681151488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50199176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37675066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483309159936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚194329681151488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191846325319710 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190846283962368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚4949 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194329576082944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191846222131764 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12701178 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3483397189120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 $ _timestamp‚1.784498906634612e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87874244 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190846266923008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185880533139968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188373807807584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483309159936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182295773574144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37675068 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50189019 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3584759565824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87864085 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37675066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482)Þ¯²2ú- _wandbruntime‚1980  _runtime‚1980ÀÔ"#û€ 2loss/FBt2/assignment_loss‚0.017326774277857875 /loss/FRt1/detection_loss‚0.5498906769844963 /loss/FBt2/detection_loss‚0.7989000608795004 &loss/total_loss‚10.734065196663284 trainer/global_step‚4949 $ _timestamp‚1.784498906635538e+09 _step‚401 /loss/FRt1/assignment_loss‚2.677943180919739 0loss/FRt2/assignment_loss‚0.9942694496024739 2loss/SRqqt1/assignment_loss‚0.6627858695178445 2loss/SRqqt2/assignment_loss‚0.6762165584866647 2loss/FBt1/assignment_loss‚0.021713827683398718 /loss/FRt2/detection_loss‚1.3447914001290928 1loss/SRqqt2/detection_loss‚0.7304144117648762 1loss/SRqqt1/detection_loss‚1.0861856860224328 .loss/FBt1/detection_loss‚1.173627300394906 epoch‚6 _runtime‚ 1980.54612436¬e^ ü… 2loss/FBt2/assignment_loss‚0.017326774277857875 /loss/FRt1/detection_loss‚0.5498906769844963 /loss/FBt2/detection_loss‚0.7989000608795004 &loss/total_loss‚10.734065196663284 trainer/global_step‚4949 $ _timestamp‚1.784498906635538e+09 _step‚401 /loss/FRt1/assignment_loss‚2.677943180919739 0loss/FRt2/assignment_loss‚0.9942694496024739 2loss/SRqqt1/assignment_loss‚0.6627858695178445 2loss/SRqqt2/assignment_loss‚0.6762165584866647 2loss/FBt1/assignment_loss‚0.021713827683398718 /loss/FRt2/detection_loss‚1.3447914001290928 1loss/SRqqt2/detection_loss‚0.7304144117648762 1loss/SRqqt1/detection_loss‚1.0861856860224328 .loss/FBt1/detection_loss‚1.173627300394906 epoch‚6 _runtime‚ 1980.54612436‘02ÔTý‚(:É é•õÒŽ×+ disk.nvme0c0n1.in‚ 0.0234375( disk.nvme0c0n1.out‚10267.37060546875& disk.nvme0n1.out‚10267.37060546875# disk.nvme1n1.out‚7303.443359375! disk.dm-0.out‚16116.333984375" disk.dm-1.out‚1333.38330078125 proc.cpu.threads‚17' disk./.usagePercent‚75.981335685766% disk./.usageGB‚1166.5034561157227% disk.nvme1c1n1.out‚7303.443359375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 56.61328125 disk.dm-0.in‚ 0.0234375 cpu‚26.335445683944958 network.sent‚ 6.559205e+06 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 193.92578125* proc.memory.percent‚17.228245735168457 network.recv‚611323 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.0234375# memory_percent‚98.8163709640503# proc.memory.rssMB‚ 2822.67578125‡eóšþ‚(: é•õÒÑѬ, gpu.0.gpu‚63.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.389' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.12966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0: ¥j2ÿ- _wandbruntime‚1999  _runtime‚1999¯¡Æ€À _runtime‚1999.121166248 _step‚402 & lr-AdamW/pg1‚9.891202756954457e-05 & lr-AdamW/pg2‚9.891202756954457e-05 trainer/global_step‚4999 $ _timestamp‚1.784498925210256e+09mðQÀËÅ trainer/global_step‚4999 $ _timestamp‚1.784498925210256e+09 _runtime‚1999.121166248 _step‚402 & lr-AdamW/pg1‚9.891202756954457e-05 & lr-AdamW/pg2‚9.891202756954457e-05’¾j#‚2‚- _wandbruntime‚1999  _runtime‚1999“”A#êƒÎO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187664038002176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3619176819200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3516568792064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12822624 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚403 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_start/reserved_Ï‹Jê"bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50675873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3516656814080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196193787709440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184044836279296 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193686106975968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3505671035978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34839029 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190180522088464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3505584887504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22016667 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187664038002176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3619176819200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3516568792064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12822624 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚403 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50675873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3516656814080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196193787709440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184044836279296 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193686106975968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3505671035978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34839029 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190180522088464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3505584887504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22016667 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34845796 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚4999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3506260838948 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88737046 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192714585284608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187700228208640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184080556340224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50685771 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192714602323968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38041118 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50695928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193723758183118 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187700429324288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192714585284608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88737046 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196231937254912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3517334930944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190217394156224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3619696772096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38041116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88726887 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3517334930944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0pÃ: ‡ÝM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184080556340224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50685771 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192714602323968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38041118 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50695928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193723758183118 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187700429324288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMo2*æÙnitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192714585284608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88737046 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196231937254912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3517334930944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190217394156224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3619696772096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38041116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88726887 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3517334930944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190217411195584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196231832186368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22020954 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚404 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38041118 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88726887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22020957 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34845796 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚4999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3506260838948 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88737046 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192714585284608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187700228208640”Ç_æ2ˆ- _wandbruntime‚1999  _runtime‚1999<׉ _step‚405 2loss/FBt2/assignment_loss‚0.009800269331507431 /loss/FRt2/detection_loss‚1.4322016678602578 0loss/SRqqt2/detection_loss‚0.743051775369767 trainer/global_step‚4999 % _timestamp‚1.7844989254699664e+09 _runtime‚1999.380613759 /loss/FRt1/detection_loss‚0.5843035253799341 0loss/FBt1/detection_loss‚0.48799715390721343 2loss/SRqqt2/assignment_loss‚0.6673068069833212 1loss/SRqqt1/detection_loss‚1.2821129354219885 &loss/total_loss‚10.467337365881935 /loss/FRt1/assignment_loss‚2.630459364194537 0loss/FRt2/assignment_loss‚1.2315089543373576 2loss/SRqqt1/assignment_loss‚0.6934291595444083 0loss/FBt1/assignment_loss‚0.2162656753242735 /loss/FBt2/detection_loss‚0.4889000782273698 epoch‚6¹êúV Š† 2loss/SRqqt2/assignment_loss‚0.6673068069833212 1loss/SRqqt1/detection_loss‚1.2821129354219885 &loss/total_loss‚10.467337365881935 /loss/FRt1/assignment_loss‚2.630459364194537 0loss/FRt2/assignment_loss‚1.2315089543373576 2loss/SRqqt1/assignment_loss‚0.6934291595444083 0loss/FBt1/assignment_loss‚0.2162656753242735 /loss/FBt2/detection_loss‚0.4889000782273698 epoch‚6 _step‚405 2loss/FBt2/assignment_loss‚0.009800269331507431 /loss/FRt2/detection_loss‚1.4322016678602578 0loss/SRqqt2/detection_loss‚0.743051775369767 trainer/global_step‚4999 % _timestamp‚1.7844989254699664e+09 _runtime‚1999.380613759 /loss/FRt1/detection_loss‚0.5843035253799341 0loss/FBt1/detection_loss‚0.48799715390721343•_ÍU‹‚(:Ê ø•õÒŽâÜ+ disk.nvme1n1.in‚0 disk.md0.out‚ 57.0390625" disk.dm-1.out‚1333.48486328125 disk.nvme1c1n1.in‚0$ memory_percent‚99.21188354492188# proc.memory.rssMB‚ 2822.78515625 network.sent‚ 6.634194e+06 disk.md0.in‚0# disk.dm-0.out‚16415.61376953125 disk.rbd0.in‚0' disk.nvme1c1n1.out‚7347.92919921875 network.recv‚615508% disk.nvme1n1.out‚7347.92919921875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.0234375 disk.nvme0n1.in‚ 0.0234375 proc.cpu.threads‚17) disk./.usagePercent‚75.98055324110297 disk.dm-0.in‚ 0.0234375( disk.nvme0c0n1.out‚10523.17529296875& disk.nvme0n1.out‚10523.17529296875# proc.memory.availableMB‚129.125) proc.memory.percent‚17.22891330718994 cpu‚26.33246309300349% disk./.usageGB‚1166.4914436340332å݃õ›Œ‚(: ø•õÒÅÁ„, gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.709' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.236333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i¡2- _wandbruntime‚2018  _runtime‚2018_íÓ>ÇŽÁ trainer/global_step‚5049 % _timestamp‚1.7844989441248121e+09 _runtime‚2018.035752714 _step‚406 & lr-AdamW/pg1‚9.884450211552755e-05 & lr-AdamW/pg2‚9.884450211552755e-05mÁ HÌÆ & lr-AdamW/pg1‚9.884450211552755e-05 & lr-AdamW/pg2‚9.884450211552755e-05 trainer/global_step‚5049 % _timestamp‚1.7844989441248121e+09 _runtime‚2018.035752714 _step‚406––‰üå2- _wandbruntime‚2018  _runtime‚2018QðbÓ'‘ÍO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51172625 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198096148874240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3550594555904 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚2018.040402323 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194545554318336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3654150243840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194545537278976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3550594555904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198096043812864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22231164 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3539500511386 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3550506533888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22231167 trainer/global_step‚5049 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539414362912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38399845 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185829619045376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÖØo Ø'’ÒO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89572470 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192024125476464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784498944125676e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198096043812864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚407 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22231164 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3539500511386 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3550506533888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22231167 trainer/global_step‚5049 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539414362912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38399845 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185829619045376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51172625 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198096148874240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3550594555904 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚2018.040402323 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194545554318336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3654150243840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194545537278976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3550594555904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198096043812864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89599848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844989443847542e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51182523 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12948774 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185865314202624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185865289298944 OADeviceStatsMonitor.on_train_batch_eå7‚"nd/allocated_bytes.all.current‚ 105068544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3551272672768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3654656516096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89599848 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194582903646208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3551184643584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22235457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189520155319296 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚5049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189519945815040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134193358336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3551184643584 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89589689 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35183975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89589689 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38407168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚195601191046526 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192060997544224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89599848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7844989443847542e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51182523 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12948774 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185865314202624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185865289298944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3551272672768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3654656516096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89599848 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194582903646208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3551184643584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22235457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189520155319296 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚5049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189519945815040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134193358336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3551184643584 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89589689 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35183975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89589689 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38407168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚195601191046526 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192060997544224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13069645 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199998404977664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197441075890618 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199998299916288 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38765897 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51669377 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90435272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196413872679936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35515309 trainer/global_step‚5099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51679520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193867745903824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844989626955702e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3689115978240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51679520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187614376907776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199998299916288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3584444275712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196413855640576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90445417 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13069645 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199998404977664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.larr¤üj/!ge_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197441075890618 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199998299916288 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38765897 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51669377 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90435272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196413872679936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35515309 trainer/global_step‚5099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51679520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193867745903824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844989626955702e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3689115978240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51679520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187614376907776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199998299916288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3584444275712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196413855640576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90445417 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35521984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187650047161344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187650072065024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193904617971584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚200036344393216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13072030 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193904600932224 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197478520721988 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚200036449461760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196451222007808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51679275 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13072292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90452491 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90462650 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22449954 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3689635635200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197478623909934 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3585122385408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38773218 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191339682796544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51679275 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90452491 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90462650 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197478520721988 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚200036449461760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196451222007808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51679275 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13072292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90452491 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90462650 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22449954 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3689635635200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197478623909934 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3585122385408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38773218 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191339682796544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51679275 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90452491 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90462650 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35521984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187650047161344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187650072065024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193904617971584œ@X ï2¤- _wandbruntime‚2036  _runtime‚2036_Ëo¥ /loss/FRt2/assignment_loss‚1.005465827290974 2loss/SRqqt1/assignment_loss‚0.7808239367501489 2loss/SRqqt2/assignment_loss‚0.6605956072284934 /loss/FBt2/detection_loss‚0.8515850180370637 /loss/FRt1/assignment_loss‚2.847391923660772 1loss/FBt2/assignment_loss‚0.00665659137824793 /loss/FRt1/detection_loss‚0.5992258756961842 /loss/FRt2/detection_loss‚1.4562194730868445 1loss/SRqqt2/detection_loss‚0.6661261896214343 /loss/FBt1/detection_loss‚2.5812825207093346 &loss/total_loss‚12.738187178135842 epoch‚6 trainer/global_step‚5099 2loss/FBt1/assignment_loss‚0.014123552969286786 1loss/SRqqt1/detection_loss‚1.2686906617070581 % _timestamp‚1.7844989629525273e+09 _runtime‚2036.863151616 _step‚413¹OB% ¦† &loss/total_loss‚12.738187178135842 epoch‚6 trainer/global_step‚5099 2loss/FBt1/assignment_loss‚0.014123552969286786 1loss/SRqqt1/detection_loss‚1.2686906617070581 % _timestamp‚1.7844989629525273e+09 _runtime‚2036.863151616 _step‚413 /loss/FRt2/assignment_loss‚1.005465827290974 2loss/SRqqt1/assignment_loss‚0.7808239367501489 2loss/SRqqt2/assignment_loss‚0.6605956072284934 /loss/FBt2/detection_loss‚0.8515850180370637 /loss/FRt1/assignment_loss‚2.847391923660772 1loss/FBt2/assignment_loss‚0.00665659137824793 /loss/FRt1/detection_loss‚0.5992258756961842 /loss/FRt2/detection_loss‚1.4562194730868445 1loss/SRqqt2/detection_loss‚0.6661261896214343 /loss/FBt1/detection_loss‚2.5812825207093346m˜Ý§‚(:„ ––õÒê©Ê, gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚114.846' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò¥«M¨‚(: ––õÒÏÈ- disk.md0.out‚57.5625$ memory_percent‚99.21233654022217 disk.md0.in‚0$ disk./.usageGB‚1166.583080291748 disk.nvme0c0n1.in‚ 0.0234375& disk.nvme0c0n1.out‚11024.099609375$ disk.nvme0n1.out‚11024.099609375" disk.dm-0.out‚17122.3857421875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.0234375& disk.nvme1c1n1.out‚7555.2724609375$ disk.nvme1n1.out‚7555.2724609375 disk.rbd0.out‚0( proc.memory.availableMB‚ 129.05078125) proc.memory.percent‚17.22891330718994 cpu‚26.326737441483314 network.sent‚ 6.745866e+06 network.recv‚624173) disk./.usagePercent‚75.98652208381358 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.0234375 disk.dm-1.out‚ 1333.83984375# proc.memory.rssMB‚ 2822.78515625 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0èu *S©‚(:È ¥–õÒµ†·+) proc.memory.percent‚17.22891330718994 network.recv‚628520" disk.dm-0.out‚17122.8583984375' proc.memory.availableMB‚ 128.9453125 proc.cpu.threads‚17 network.sent‚ 6.808866e+06) disk./.usagePercent‚75.98652730176782% disk./.usageGB‚1166.5831604003906% disk.nvme0c0n1.out‚11025.01953125# disk.nvme0n1.out‚11025.01953125 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚7556.02197265625 disk.md0.in‚0 disk.nvme0c0n1.in‚ 0.0234375 disk.nvme0n1.in‚ 0.0234375' disk.nvme1c1n1.out‚7556.02197265625 disk.nvme1n1.in‚0 disk.md0.out‚ 58.0234375 disk.dm-0.in‚ 0.0234375 disk.rbd0.in‚0 cpu‚26.323786296153337 disk.dm-1.in‚0" disk.dm-1.out‚1334.04833984375 disk.rbd0.out‚0$ memory_percent‚99.21298027038574# proc.memory.rssMB‚ 2822.78515625;Ìçišª‚(: ¥–õÒñ×à+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.709' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.236333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•JFÂ2«- _wandbruntime‚2055  _runtime‚20557Á’ÑE¬Á _step‚414 & lr-AdamW/pg1‚9.870714749299163e-05 & lr-AÛE캂damW/pg2‚9.870714749299163e-05 trainer/global_step‚5149 % _timestamp‚1.7844989816770132e+09 _runtime‚2055.587933964ÁÚÉÔÌ­Æ _step‚414 & lr-AdamW/pg1‚9.870714749299163e-05 & lr-AdamW/pg2‚9.870714749299163e-05 trainer/global_step‚5149 % _timestamp‚1.7844989816770132e+09 _runtime‚2055.587933964ž Óí2®- _wandbruntime‚2055  _runtime‚2055›vØçÔ'¯ÎO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198282174002176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39131947 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39131945 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189399109866496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199318405566192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13193548 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39131947 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195711349291824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52166129 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3618470039552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3607073313728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201900556019712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193123228934144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618382017536 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3724094163968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22660167 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3723907459072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚415 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 Ô¯bÙ'°ÓO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13193548 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39131947 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195711349291824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52166129 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3618470039552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3607073313728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201900556019712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193123228934144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618382017536 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3724094163968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22660167 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3723907459072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚415 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198282174002176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39131947 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39131945 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189399109866496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199318405566192Ÿ‘£ß>2±- _wandbruntime‚2055  _runtime‚2055°¶…Þ&²ØM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39139266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3724605511168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199355953585396 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195748221359584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3619148156416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201938600496640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198319540369408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198319557408768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35860403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22664457 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39139268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39139268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193159410534912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189434829927424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13195946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2055.848344578 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3619060127232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201938705565184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91315293 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52186184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35860141 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198319557408768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3619060127232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189434805023744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844989819372466e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91315293 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198319540369408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199356056773342 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚5149 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52176027 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91325452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195687 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195748204320224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193159622136320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39139266 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91325452Ï70‹³ÝM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91325452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195687 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195748204320224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193159622136320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39139266 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91325452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39139266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3724605511168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199355953585396 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195748221359584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3619148156416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitšPïXor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201938600496640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198319540369408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198319557408768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35860403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22664457 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39139268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39139268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193159410534912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189434829927424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13195946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2055.848344578 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3619060127232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201938705565184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91315293 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52186184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35860141 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198319557408768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3619060127232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189434805023744 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844989819372466e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91315293 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198319540369408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199356056773342 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚5149 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52176027 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EPfR·‚(:Ç ´–õÒš‡¯+ disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.0234375 disk.rbd0.out‚0" disk.nvme1n1.out‚ 7557.62109375$ memory_percent‚99.21338558197021# proc.memory.rssMB‚ 2822.78515625) proc.memory.percent‚17.22891330718994" disk.dm-1.out‚1334.13818359375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚11030.6630859375 disk.md0.in‚0% disk.nvme0n1.out‚11030.6630859375 disk.nvme1n1.in‚0% disk./.usageGB‚1166.5836181640625 disk.nvme0c0n1.in‚ 0.0234375 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 7557.62109375( proc.memory.availableMB‚ 128.87890625 cpu‚26.32109642271259 proc.cpu.threads‚17 network.sent‚ 6.811788e+06 network.recv‚631794) disk./.usagePercent‚75.98655711864924 disk.md0.out‚ 58.3828125 disk.dm-0.in‚ 0.0234375# disk.dm-0.out‚17129.22705078125Ð÷ç¡š¸‚(: ´–õÒæˇ, gpu.0.gpu‚83.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.95' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.98333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¦ª[2¹- _wandbruntime‚2074  _runtime‚2074¼ß}ĺ¾ % lr-AdamW/pg1‚9.86373205244858e-05 % lr-AdamW/pg2‚9.86373205244858e-05 trainer/global_step‚5199 $ _timestamp‚1.784499000641284e+09 _runtime‚2074.553410742 _step‚418^Ad“ɻà % lr-AdamW/pg1‚9.86373205244858e-05 % lr-AdamW/pg2‚9.86373205244858e-05 trainer/global_step‚5199 $ _timestamp‚1.784499000641284e+09 _runtime‚2074.553410742 _step‚418¢DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197554952679824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39497995 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚419 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22874664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200150509403136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201195941617434 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194942745037312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191183867728896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36192533 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203802812123136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844990006418462e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52673024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3652407781376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3652319759360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36192288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22874667 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52673024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52662881 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3759055624704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3758877308416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194942948257280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203802917184512 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92160876 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92171021 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39497995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52662881 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39497997 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200150492363776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13317624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203802812123136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844990006418462e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52673024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3652407781376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3652319759360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36192288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22874667 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52673024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52662881 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3759055624704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3758877308416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194942948257280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203802917184512 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92160876 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92171021 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39497995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52662881 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39497997 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200150492363776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13317624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197554952679824 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39497995 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚419 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22874664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200150509403136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201195941617434 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194942745037312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191183867728896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36192533 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303£HH˜Î2¿- _wandbruntime‚2074  _runtime‚2074lZäCÀØM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52682936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52672779 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22878957 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191219562886144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92178095 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requeste”–úS›#d_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203840856600064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39505318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13320302 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200187858731008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194979135944704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚194979347546112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844990009019823e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203840961668608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚2074.813064844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3652997869056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3759573058560 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92188254 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203840961668608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3653085898240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191219587789824 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36199006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92188254 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39505316 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92178095 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3653085898240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36199259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22878954 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3641664889166ò®ð—ã&ÁÝM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92178095 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3653085898240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36199259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22878954 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3641664889166 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52682936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52672779 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22878957 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191219562886144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92178095 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203840856600064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39505318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13320302 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200187858731008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194979135944704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚194979347546112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844990009019823e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203840961668608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚2074.813064844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3652997869056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3759573058560 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92188254 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203840961668608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3653085898240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191219587789824 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36199006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92188254 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39505316 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¤;¥ Í2Â- _wandbruntime‚2074  _runtime‚20742s–sÃÿ /loss/FBt2/detection_loss‚0.5470995107135687 epoch‚6 0loss/FRt2/assignment_loss‚1.0844175079669334 2loss/SRqqt1/assignment_loss‚0.9255951476112312 /loss/FBt1/detection_loss‚0.6671668795276094 _runtime‚2074.813574975 _step‚421 2loss/SRqqt2/assignment_loss‚0.6101791540647603 1loss/FBt1/assignment_loss‚0.16196859839162803 2loss/FBt2/assignment_loss‚0.005473385126841153 1loss/SRqqt2/detection_loss‚0.6872695134782157 % _timestamp‚1.7844990009029572e+09 ,loss/FRt1/assignment_loss‚2.857019706877 &loss/total_loss‚10.872440376846384 trainer/global_step‚5199 /loss/FRt1/detection_loss‚0.6310775536008688 /loss/FRt2/detection_loss‚1.4013032477402838 1loss/SRqqt1/detection_loss‚1.2938701717474428˜àÿ Ä„ 2loss/SRqqt1/assignment_loss‚0.9255951476112312 /loss/FBt1/detection_loss‚0.6671668795276094 _runtime‚2074.813574975 _step‚421 2loss/SRqqt2/assignment_loss‚0.6101791540647603 1loss/FBt1/assignment_loss‚0.16196859839162803 2loss/FBt2/assignment_loss‚0.005473385126841153 1loss/SRqqt2/detection_loss‚0.6872695134782157 % _timestamp‚1.7844990009029572e+09 ,loss/FRt1/assignment_loss‚2.857019706877 &loss/total_loss‚10.872440376846384 trainer/global_step‚5199 /loss/FRt1/detection_loss‚0.6310775536008688 /loss/FRt2/detection_loss‚1.4013032477402838 1loss/SRqqt1/detection_loss‚1.2938701717474428 /loss/FBt2/detection_loss‚0.5470995107135687 epoch‚6 0loss/FRt2/assignment_loss‚1.0844175079669334¥toRÅ‚(:Ç ÖõÒùÓÛ+! disk.dm-1.out‚1334.2607421875 disk.nvme0c0n1.in‚ 0.0234375 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2822.78515625 proc.cpu.threads‚17 disk.md0.out‚58.625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11031.40869140625 disk.nvme0n1.in‚ 0.0234375& disk.nvme0n1.out‚11031.40869140625$ memory_percent‚99.21364784240723' proc.memory.availableMB‚ 128.8359375) proc.memory.percent‚17.22891330718994) disk./.usagePercent‚75.98655587627918 disk.md0.in‚0 disk.dm-0.in‚ 0.0234375# disk.dm-0.out‚17129.75439453125& disk.nvme1c1n1.out‚7558.0712890625$ disk.nvme1n1.out‚7558.0712890625 cpu‚26.31819592270356 network.sent‚ 6.887878e+06 network.recv‚636045 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0% disk./.usageGB‚1166.5835990905762 disk.dm-1.in‚0K ü­Æ‚(:„ ÖõÒ, gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚187.248' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚62.416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©.Î 2Ç- _wandbruntime‚2093  _runtime‚2093i¡&§ÇÈÁ & lr-AdamW/pg1‚9.856672860495884e-05 & lr-AdamW/pg2‚9.856672860495884e-05 trainer/global_step‚5249 % _timestamp‚1.7844990195782776e+09 _runtime‚2093.489256257 _step‚422ì¶yÜÌÉÆ _step‚422 & lr-AdamW/pg1‚9.856672860495884e-05 & lr-AdamW/pg2‚9.856672860495884e-05 trainer/global_step‚5249 % _timestamp‚1.7844990195782776e+09 _runtime‚2093.489256257¦+Ú‚^2Ê- _wandbruntime‚2093  _runtime‚20937ÇÔ'ËÎO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3674818413018 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39864045 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93023678 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39864047 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3686345523200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13440974 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3793971409408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚423 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202018810725376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚203073271293008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53159633 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205705068226560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23089164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205705068226560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202018827764736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23089167 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3686257501184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93033823 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13440714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844990195792768e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36530141 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192968625591296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203073374480842 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205705173287936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192968650494976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39864047 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199398556067824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202018827764736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196762418684416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚5249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3674732264544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53159633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196762621904384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3686345523200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205705173287936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3686257501184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202018810725376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36529878 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3793793093120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39864045 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202018810725376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36529878 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3793793093120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39864045 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205705068226560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23089164 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205705068226560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202018827764736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23089167 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3686257501184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93033823 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13440714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844990195792768e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36530141 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192968625591296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203073374480842§ÐèD2Í- _wandbruntime‚2093  _runtime‚2093fÁûVÜ&ÎÖM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205743217772032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199435428135584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3675494364574 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205743217772032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193004320748544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3675408215988 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93051056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196798814194688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93051056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93040897 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202056194131968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205743112703488 _step‚424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39871366 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _runtime‚ 2093.7494778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23093454 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39871366 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53169531 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39871368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53179688 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3794673852416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53169531 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193004345652224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202056194131968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53179688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39871368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3687023640064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202056177092608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3687023640064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 % _timestamp‚1.7844990198383873e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23093457 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36536570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13443116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93040897 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3686935610880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36536814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3794493446144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0VÝ„¶á&ÏÛM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196798814194688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93051056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93040897 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202056194131968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205743112703488 _step‚424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39871366 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _runtime‚ 2093.7494778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23093454 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39871366 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53169531 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39871368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53179688 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3794673852416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53169531 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193004345652224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202056194131968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53179688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39871368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3687023640064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202056177092608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3687023640064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 % _timestamp‚1.7844990198383873e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23093457 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36536570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13443116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93040897 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3686935610880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36536814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3794493446144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205743217772032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199435428135584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3675494364574 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205743217772032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193004320748544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3675408215988 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93051056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330¨/3¢2Ð- _wandbruntime‚2093  _runtime‚20932¤díÑÿ 0loss/FRt1/assignment_loss‚2.5833840697934622 0loss/FBt1/assignment_loss‚0.0461135466458226 /loss/FRt2/detection_loss‚1.2721511046665155 /loss/FBt1/detection_loss‚0.8993557432873381 % _timestamp‚1.7844990198393192e+09 _step‚425 2loss/SRqqt1/assignment_loss‚0.8956385597135614 1loss/SRqqt1/detection_loss‚1.2252315221202073 0loss/SRqqt2/detection_loss‚0.661463816719821 &loss/total_loss‚10.753301867884673 epoch‚6 _runtime‚2093.749914541 2loss/FBt2/assignment_loss‚0.006134365674705278 .loss/FRt1/detection_loss‚0.630406755298617 /loss/FBt2/detection_loss‚0.7130229846338851 /loss/FRt2/assignment_loss‚1.084356366993977 2loss/SRqqt2/assignment_loss‚0.7360430323367593 trainer/global_step‚5249U;ùà Ò„ 0loss/SRqqt2/detection_loss‚0.661463816719821 &loss/total_loss‚10.753301867884673 epoch‚6 _runtime‚2093.749914541 2loss/FBt2/assignment_loss‚0.006134365674705278 .loss/FRt1/detection_loss‚0.630406755298617 /loss/FBt2/detection_loss‚0.7130229846338851 /loss/FRt2/assignment_loss‚1.084356366993977 2loss/SRqqt2/assignment_loss‚0.7360430323367593 trainer/global_step‚5249 0loss/FRt1/assignment_loss‚2.5833840697934622 0loss/FBt1/assignment_loss‚0.0461135466458226 /loss/FRt2/detection_loss‚1.2721511046665155 /loss/FBt1/detection_loss‚0.8993557432873381 % _timestamp‚1.7844990198393192e+09 _step‚425 2loss/SRqqt1/assignment_loss‚0.8956385597135614 1loss/SRqqt1/detection_loss‚1.2252315221202073©âR”«VÓ‚(:Ë Ò–õÒøáá+" disk.nvme1n1.out‚ 7566.91015625 disk.md0.out‚ 59.12109375! disk.dm-1.out‚1334.3701171875 proc.cpu.threads‚17 network.recv‚640641) disk./.usagePercent‚75.98628876671654 disk.nvme0n1.in‚ 0.02734375$ disk.nvme1c1n1.out‚ 7566.91015625$ memory_percent‚99.21393394470215' proc.memory.availableMB‚ 128.7890625% disk./.usageGB‚1166.5794982910156 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.02734375 disk.md0.in‚0 disk.dm-0.in‚ 0.02734375# disk.dm-0.out‚17139.38916015625# proc.memory.rssMB‚ 2822.78515625) proc.memory.percent‚17.22891330718994 network.sent‚ 6.94912e+06 disk.rbd0.in‚0 disk.dm-1.in‚0 cpu‚26.315344276301204( disk.nvme0c0n1.out‚11033.35888671875& disk.nvme0n1.out‚11033.35888671875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0ᇗœÔ‚(:Ž Ò–õÒûàˆ, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 g"!¡wýpu.0.temp‚47.0 gpu.0.powerWatts‚199.925' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.64166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê,”2Õ- _wandbruntime‚2112  _runtime‚2112­ãfÆÖÀ & lr-AdamW/pg2‚9.849537286508366e-05 trainer/global_step‚5299 % _timestamp‚1.7844990382145956e+09 _runtime‚ 2112.12737989 _step‚426 & lr-AdamW/pg1‚9.849537286508366e-05%N¹Ë×Å _runtime‚ 2112.12737989 _step‚426 & lr-AdamW/pg1‚9.849537286508366e-05 & lr-AdamW/pg2‚9.849537286508366e-05 trainer/global_step‚5299 % _timestamp‚1.7844990382145956e+09ªhØ]ê2Ø- _wandbruntime‚2112  _runtime‚2112×ìÒ'ÙÌO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3708561739952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207607429391360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3720195243008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚ 2112.1274426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40230097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36868302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40230095 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207607324329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53656385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23303664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204950704156416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93896625 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23303667 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204950807344250 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚203887146126336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚427 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53666528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53656385 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13564635 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207607429391360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207607324329984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40230095 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53666528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194753383453696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201242159455824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198582130379264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201242142416464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93886480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203887129086976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚5299 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3828925241856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3828746925568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0œ¯¤×'ÚÑO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3708561739952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207607429391360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3720195243008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚ 2112.1274426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40230097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36868302 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40230095 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207607324329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53656385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23303664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204950704156416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93896625 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23303667 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204950807344250 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚203887146126336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚427 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53666528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53656385 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13564635 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207607429391360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207607324329984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40230095 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53666528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194753383453696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201242159455824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198582130379264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201242142416464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93886480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203887129086976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚5299 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3828925241856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3828746925568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0«¯Ä 2Û- _wandbruntime‚2112  _runtime‚2112—×NÞ&ÜØM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203924512493568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚203924512493568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93913858 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204988252175620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3720961381888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3720961381888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40237416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40237418 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53666283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93903699 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201279014484224 trainer/global_step‚5299 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36874802 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3709323839982 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7844990384746683e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207645473875456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13566848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93903699 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3720873352704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194789103514624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829437969408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23307954 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93913858 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53676440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222087168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚428 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53676440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207645368806912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203924495454208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40237416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194789078610944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829635152896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3709237691396 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544ÐKÄŽÝÝM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36874802 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3709323839982 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7844990384746683e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207645473875456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚oÎè—Ô 13566848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93903699 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3720873352704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194789103514624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829437969408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23307954 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93913858 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53676440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222087168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚428 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53676440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207645368806912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203924495454208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40237416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194789078610944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829635152896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3709237691396 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203924512493568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚203924512493568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93913858 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204988252175620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3720961381888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3720961381888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40237416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40237418 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53666283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93903699 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201279014484224 trainer/global_step‚5299 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205755464487936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3754221006848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3863675615232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3742477363834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209509685494784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205755447448576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40596147 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13688479 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203085762843824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54153137 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚196538141316096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚2131.035438544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚205755447448576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3754132984832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206828240207658 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209509580433408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚431 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 I–îä$êÓO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3754132984832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206828137019824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40596145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 trainer/global_step‚5349 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54153137 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3754221006848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚196538166219776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54163280 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40596145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54163280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209509685494784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3742391215360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7844990571227493e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37206905 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209509580433408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23518164 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13688738 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203085745804464 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94749282 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94759427 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40596147 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200401816931328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205755464487936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3754221006848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3863675615232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3742477363834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209509685494784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205755447448576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40596147 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13688479 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203085762843824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54153137 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚196538141316096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚2131.035438544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚205755447448576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3754132984832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206828240207658 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209509580433408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚431 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚205792813815808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3754899123712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94776660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209547624910336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54173192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54163035 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3754811094528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94776660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40603468 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205792830855168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23522457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200438423024128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚196573861377024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209547729978880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37213417 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844990573830855e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23522454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196573836473344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54173192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206865788226974 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209547729978880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203122634911584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203122617872224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94766501 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40603466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40603468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3743067166804 _runtime‚2131.294157335 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205792830855168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3743153315390 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13691222 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94766501 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3754899123712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205792813815808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3864379143680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0~]¢]ã&íÝM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40603468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3743067166804 _runtime‚2131.294157335 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205792830855168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3743153315390 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13691222 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94766501 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3754899123712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205792813815808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3864379143680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚205792813815808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3754899123712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94776660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209547624910336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54173192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54163035 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3754811094528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94776660 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40603468 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205792830855168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23522457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200438423024128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚196573861377024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209547729978880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37213417 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844990573830855e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23522454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196573836473344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54173192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206865788226974 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209547729978880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203122634911584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203122617872224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94766501 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40603466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152°É+Œí2î- _wandbruntime‚2131  _runtime‚2131Žå½aïÿ /loss/FRt1/detection_loss‚0.5837678520811286 $loss/total_loss‚10.9532125810881 _step‚433 0loss/FRt1/assignment_loss‚2.8654289957619112 1loss/FBt1/assignment_loss‚0.11132061858977375 .loss/FRt2/detection_loss‚1.425562997567549 epoch‚6 trainer/global_step‚5349 % _timestamp‚1.7844990573839917e+09 2loss/SRqqt1/assignment_loss‚0.8068028589321931 2loss/FBt2/assignment_loss‚0.010001769504145918 0loss/SRqqt1/detection_loss‚1.256325452779163 1loss/SRqqt2/detection_loss‚0.5624551919303755 /loss/FBt2/detection_loss‚0.6148164941891029 _runtime‚2131.294621286 0loss/FRt2/assignment_loss‚1.1060166959176985 2loss/SRqqt2/assignment_loss‚0.5781142482081977 /loss/FBt1/detection_loss‚1.0325994056268615Ïe^È ð„ 0loss/FRt2/assignment_loss‚1.1060166959176985 2loss/SRqqt2/assignment_loss‚0.5781142482081977 /loss/FBt1/detection_loss‚1.0325994056268615 /loss/FRt1/detection_loss‚0.5837678520811286 $loss/total_loss‚10.9532125810881 _step‚433 0loss/FRt1/assignment_loss‚2.8654289957619112 1loss/FBt1/assignment_loss‚0.11132061858977375 .loss/FRt2/detection_loss‚1.425562997567549 epoch‚6 trainer/global_step‚5349 % _timestamp‚1.7844990573839917e+09 2loss/SRqqt1/assignment_loss‚0.8068028589321931 2loss/FBt2/assignment_loss‚0.010001769504145918 0loss/SRqqt1/detection_loss‚1.256325452779163 1loss/SRqqt2/detection_loss‚0.5624551919303755 /loss/FBt2/detection_loss‚0.6148164941891029 _runtime‚2131.294621286±ü¥hOñ‚(:Ä ÿ–õÒ±Þà+ disk.dm-0.in‚ 0.02734375 disk.dm-0.out‚17153.16015625 disk.nvme0n1.in‚ 0.02734375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚7575.177734375# disk.nvme1n1.out‚7575.177734375 disk.md0.out‚ 60.21484375 disk.rbd0.in‚0$ disk.nvme0n1.out‚11047.974609375# proc.memory.rssMB‚ 2822.78515625) proc.memory.percent‚17.22891330718994 cpu‚26.30749479432397 network.recv‚652649$ disk./.usageGB‚1166.584114074707 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.02734375$ memory_percent‚99.21424388885498 network.sent‚ 7.078526e+06) disk./.usagePercent‚75.98658942027076 disk.nvme1n1.in‚0 disk.dm-1.out‚ 1341.11328125& disk.nvme0c0n1.out‚11047.974609375( proc.memory.availableMB‚ 128.73828125 proc.cpu.threads‚17?r°º™ò‚(:Ž ÿ–õÒ¤ÔŽ, gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.22' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.07333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)MÙ¾2ó- _wandbruntime‚2149  _runtime‚2149•-l ÇôÁ & lr-AdamW/pg2‚9.835037450813234e-05 trainer/global_step‚5399 % _timestamp‚1.7844990760778527e+09 _runtime‚2149.988781594 _step‚434 & lr-AdamW/pg1‚9.835037450813234e-05ž#£©ÌõÆ & lr-AdamW/pg1‚9.835037450813234e-05 & lr-AdamW/pg2‚9.835037450813234e-05 trainer/global_step‚5399 % _timestamp‚1.7844990760778527e+09 _runtime‚2149.988781594 _step‚434²µÔra2ö- _wandbruntime‚2149  _runtime‚2149ðŽ™q–!÷ÎO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3898836882432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207623765810176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3788158748672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95612084 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3788158748672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23732664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198322924082176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54649889 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207623765810176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _step‚435 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211411836536832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3898650177536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40962197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211411836536832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204929366231824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211411941598208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54649889 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40962197 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13812346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202221549356032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚5399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211411941598208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204929349192464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3776306839242 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207623782849536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95622229 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40962195 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208705673071066 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3776220690ÖóŒ>768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40962195 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13812098 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3788070726656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207623782849536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23732667 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152÷––Ù'øÓO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3898650177536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40962197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211411836536832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204929366231824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211411941598208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54649889 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40962197 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13812346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202221549356032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚5399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211411941598208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204929349192464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3776306839242 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207623782849536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95622229 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40962195 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208705673071066 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3776220690768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40962195 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13812098 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3788070726656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207623782849536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23732667 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3898836882432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207623765810176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3788158748672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95612084 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3788158748672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23732664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198322924082176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54649889 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207623765810176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _step‚435 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211411836536832³ߟ^Ý2ù- _wandbruntime‚2150  _runtime‚2150ûÞmLÞ&úØM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207661132177408 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95629303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207661149216768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207661132177408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54669944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844990763385532e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13814533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54659787 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40969516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211449986082304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3899531349504 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95639462 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202257949473280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3788748836352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54669944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3776982790798 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3776896642212 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37551743 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3788748836352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40969516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37551487 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23736957 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3788836865536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207661149216768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198358619239424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚5399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54659787 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95639462 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211449986082304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899355137536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208743117902436 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211449881013760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13814786 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95629303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40969518 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816îÄjã&ûÝM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207661132177408 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95629303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207661149216768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207661132177408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54669944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844990763385532e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13814533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54659787 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40969516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211449986082304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3899531349504 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95639462 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202257949473280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3788748836352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54669944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3776982790798 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3776896642212 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37551743 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3788748836352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40969516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37551487 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23736957 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3788836865536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207661149216768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198358619239424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚5399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54659787 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95639462 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211449986082304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3899355137536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208743117902436 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211449881013760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13814786 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95629303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40969518 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816´Cõ2ü- _wandbruntime‚2150  _runtime‚2150@,fý 2loss/SRqqt2/assignment_loss‚0.5797984235461665 % _timestamp‚1.7844990763394847e+09 /loss/FRt1/assignment_loss‚2.725857528785911 2loss/SRqqt1/assignment_loss‚0.7833098967961383 1loss/FBt1/assignment_loss‚0.14818934267543327 1loss/FBt2/assignment_loss‚0.06306880029218756 /loss/FRt2/detection_loss‚1.2619519497791187 1loss/SRqqt1/detection_loss‚1.0931772230424328 1loss/SRqqt2/detection_loss‚0.6262693851485273 /loss/FBt2/detection_loss‚0.6258832008704249 0loss/FRt2/assignment_loss‚1.0667526965157244 /loss/FRt1/detection_loss‚0.5939691202295079 /loss/FBt1/detection_loss‚1.1325246780183158 epoch‚6 trainer/global_step‚5399 _runtime‚2150.250113417 _step‚437 &loss/total_loss‚10.700752245699888%Úl™…þ† 2loss/SRqqt2/assignment_loss‚0.5797984235461665 % _timestamp‚1.7844990763394847e+09 /loss/FRt1/assignment_loss‚2.72u¯~µ‡5857528785911 2loss/SRqqt1/assignment_loss‚0.7833098967961383 1loss/FBt1/assignment_loss‚0.14818934267543327 1loss/FBt2/assignment_loss‚0.06306880029218756 /loss/FRt2/detection_loss‚1.2619519497791187 1loss/SRqqt1/detection_loss‚1.0931772230424328 1loss/SRqqt2/detection_loss‚0.6262693851485273 /loss/FBt2/detection_loss‚0.6258832008704249 0loss/FRt2/assignment_loss‚1.0667526965157244 /loss/FRt1/detection_loss‚0.5939691202295079 /loss/FBt1/detection_loss‚1.1325246780183158 epoch‚6 trainer/global_step‚5399 _runtime‚2150.250113417 _step‚437 &loss/total_loss‚10.700752245699888µÖItXÿ‚(:Í Ž—õÒƒõ×+ network.sent‚ 7.140623e+06 disk.md0.in‚0 disk.md0.out‚ 60.52734375 disk.nvme0c0n1.in‚ 0.02734375 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2822.9609375( proc.memory.percent‚17.2299861907959 proc.cpu.threads‚17) disk./.usagePercent‚75.98658693553065% disk./.usageGB‚1166.5840759277344# disk.dm-0.out‚17154.06982421875 disk.dm-1.in‚0 disk.dm-1.out‚ 1341.23046875$ disk.nvme1n1.out‚7575.7548828125( proc.memory.availableMB‚ 132.40234375 disk.dm-0.in‚ 0.02734375 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11049.08642578125 disk.nvme0n1.in‚ 0.02734375& disk.nvme0n1.out‚11049.08642578125& disk.nvme1c1n1.out‚7575.7548828125 cpu‚26.305143944355425 network.recv‚656657 disk.nvme1n1.in‚0$ memory_percent‚99.19188022613525fu¿€‚(:… Ž—õÒ…€, gpu.0.gpu‚76.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚131.454' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚43.818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lˆæÿ2- _wandbruntime‚2168  _runtime‚2168 ×ÖdÇ‚Á & lr-AdamW/pg1‚9.827673421349924e-05 & lr-AdamW/pg2‚9.827673421349924e-05 trainer/global_step‚5449 % _timestamp‚1.7844990947426095e+09 _runtime‚2168.655404362 _step‚438±ÄוÌƒÆ & lr-AdamW/pg2‚9.827673421349924e-05 trainer/global_step‚5449 % _timestamp‚1.7844990947426095e+09 _runtime‚2168.655404362 _step‚438 & lr-AdamW/pg1‚9.827673421349924e-05¶ÐM 2„- _wandbruntime‚2168  _runtime‚2168[ÓPâÓ'…ÍO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209492101211136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3822008468480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41328247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23947167 $ _timestamp‚1.784499094743129e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213314197701632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200107681944576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96485031 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚2168.655492813 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55146641 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204041277974016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3810136314650 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41328245 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213314092640256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209492084171776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209492101211136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209492084171776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37882890 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204041485388288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55146641 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96474886 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3933803443712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3933620933120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3810050166176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200107657040896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13935966 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213314092640256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3822096490496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206772952580464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206772969619824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206772952580464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206772969619824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37882890 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204041485388288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55146641 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96474886 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3933803443712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3933620933120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3810050166176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200107657040896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13935966 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213314092640256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3822096490496 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210620653953790 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96502264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37889644 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200143377101824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209529467578368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206809841687584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 $ _timestamp‚1.784499094991645e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209529467578368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96492105 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41335568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41335566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96502264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210620550765844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3822686578176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41335566 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209529450539008 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23951454 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3810812266206 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13938190 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3822774607360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3822774607360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55166696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55156539 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213352137117184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3810726117620 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209529450539008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200143352198144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3934320881664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23951457 LDDeviceStatsMonitor.on_train_batch_end/inactive_spli3EÈ®mt.large_pool.peak‚238 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213352242185728M-(â&‰ÜM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41335568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37889899 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209529450539008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200143352198144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3934320881664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23951457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213352242185728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213352242185728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204077878389760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3934501287936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210620653953790 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96502264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37889644 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200143377101824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209529467578368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206809841687584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 $ _timestamp‚1.784499094991645e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209529467578368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96492105 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41335568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41335566 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96502264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210620550765844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3822686578176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41335566 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209529450539008 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23951454 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3810812266206 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13938190 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3822774607360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3822774607360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55166696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55156539 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213352137117184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3810726117620 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55653536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205861203721216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97337688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚443 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211360419572736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55643393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚208616573007824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211360419572736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24161664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211360402533376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41694297 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55653536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215216453805056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215216348743680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215216453805056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212460538797882 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3843965790058Žúç<]#”ÓO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215216453805056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215216348743680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215216453805056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212460538797882 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3843965790058 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41694295 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201892414903296 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚208616555968464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3855946210304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3855946210304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41694295 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97347833 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14060336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201892439806976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212460435610048 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38222003 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205860990015488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3968763914240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55653536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205861203721216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97337688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚443 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211360419572736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55643393 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚208616573007824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211360419572736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24161664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211360402533376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41694297 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55653536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303»०„2•- _wandbruntime‚2186  _runtime‚2186¨¶}Þ&–ØM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55653291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38228779 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24165957 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215254498289152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212498086817198 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41701616 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55653291 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211397768900608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38228539 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3856712349184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41701618 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55663448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3969453677056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3969273270784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97354907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97365066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚208653445075584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211397768900608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211397785939968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856624320000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚2186.971340557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14062822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3844641741614 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14062585 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205897588641280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215254393220608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330ö‚¯ã&—ÝM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215254393220608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55653291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38228779 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24165957 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215254498289152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212498086817198 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41701616 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55653291 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211397768900608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38228539 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3856712349184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41701618 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55663448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3969453677056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3969273270784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97354907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97365066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚208653445075584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211397768900608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211397785939968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856624320000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚2186.971340557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14062822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3844641741614 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14062585 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205897588641280¼bQêú2˜- _wandbruntime‚2186  _runtime‚2186ï“ô”™ÿ 2loss/SRqqt1/assignment_loss‚0.8778340911823175 1loss/FBt1/assignment_loss‚0.22708225143554955 .loss/FRt1/detection_loss‚0.558820800316998 1loss/SRqqt2/detection_loss‚0.6196824319121477 % _timestamp‚1.7844991130612466e+09 1loss/SRqqt1/detection_loss‚1.0815848478506649 _runtime‚2186.971901538 _step‚445 &loss/total_loss‚10.871965221089699 .loss/FRt1/assignment_loss‚2.55759540545313 0loss/FRt2/assignment_loss‚0.9655167433552276 2loss/SRqqt2/assignment_loss‚0.6396767981221753 .loss/FBt2/detection_loss‚1.116717180951188 trainer/global_step‚5499 2loss/FBt2/assignment_loss‚0.014429028578373011 /loss/FRt2/detection_loss‚1.4588633380319111 /loss/FBt1/detection_loss‚0.7541623039000157 epoch‚6Š•Ì š„ 1loss/SRqqt1/detection_loss‚1.0815848478506649 _runtime‚2186.971901538 _step‚445 &loss/total_loss‚10.871965221089699 .loss/FRt1/assignment_loss‚2.55759540545313 0loss/FRt2/assignment_loss‚0.9655167433552276 2loss/SRqqt2/assignment_loss‚0.6396767981221753 .loss/FBt2/detection_loss‚1.116717180951188 trainer/global_step‚5499 2loss/FBt2/assignment_loss‚0.014429028578373011 /loss/FRt2/detection_loss‚1.4588633380319111 /loss/FBt1/detection_loss‚0.7541623039000157 epoch‚6 2loss/SRqqt1/assignment_loss‚0.8778340911823175 1loss/FBt1/assignment_loss‚0.22708225143554955 .loss/FRt1/detection_loss‚0.558820800316998 1loss/SRqqt2/detection_loss‚0.6196824319121477 % _timestamp‚1.7844991130612466e+09½ÀCØg[›‚(:Ð ¬—õÒªŸÌ+" proc.memory.rssMB‚ 2822.9609375 network.sent‚ 7.252164e+06 network.recv‚666014 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 61.17578125 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚1166.344554901123 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7577.05908203125% disk.nvme1n1.out‚7577.05908203125 disk.dm-0.in‚ 0.02734375# disk.dm-0.out‚17159.35791015625 disk.nvme0c0n1.in‚ 0.02734375( proc.memory.percent‚17.2299861907959) disk./.usagePercent‚75.97098550080594( disk.nvme0c0n1.out‚11054.80615234375 disk.nvme0n1.in‚ 0.02734375& disk.nvme0n1.out‚11054.80615234375$ memory_percent‚99.19238090515137 cpu‚26.300764089420678 proc.cpu.threads‚17" disk.dm-1.out‚1341.55615234375 disk.rbd0.out‚0' proc.memory.availableMB‚ 132.3203125í¯·ã™œ‚(:Ž ¬—õÒØ¿¡, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.908' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.30266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09O”×2- _wandbruntime‚2204  _runtime‚2204nE¶ÆžÀ & lr-AdamW/pg1‚9.812717728939317e-05 & lr-AdamW/pg2‚9.812717728939317e-05 trainer/global_step‚5549 $ _timestamp‚1.784499130581297e+09 _runtime‚2204.492206354 _step‚446SIhrËŸÅ _runtime‚2204.492206354 _step‚446 & lr-AdamW/pg1‚9.812717728939317e-05 & lr-AdamW/pg2‚9.812717728939317e-05 trainer/global_step‚5549 $ _timestamp‚1.784499130581297e+09¾}ÙiÛ2 - _wandbruntime‚2204  _runtime‚2204ƒ=ê ¡ÎO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24376167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3877709116992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42060345 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3889971974144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217118604847104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56150288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98210635 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42060347 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56140145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24376164 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210460176395824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213228737934336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217118709908480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚5549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213228720894976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_stJªQÏart/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42060347 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217118604847104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213228737934336 % _timestamp‚1.7844991305820928e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213228720894976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚123030‚ÁÖÙ'¢ÓO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42060345 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3889971974144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217118604847104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56150288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98210635 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42060347 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56140145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24376164 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210460176395824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213228737934336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217118709908480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚5549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213228720894976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42060347 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217118604847104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213228737934336 % _timestamp‚1.7844991305820928e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213228720894976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24376167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3877709116992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482¿ ®ð‘2£- _wandbruntime‚2204  _runtime‚2204F–´Þ&¤ØM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98227868 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56150043 trainer/global_step‚5549 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14185407 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3878385068436 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56160200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207717087588352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213266087262208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3878471217022 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38566100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214375416492660 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98217709 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213266104301568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213266087262208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203712892826624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42067666 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213266104301568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14185643 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214375519680606 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98217709 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56150043 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38565861 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42067668 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3890562061824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24380454 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ú(„.ã&¥ÝM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56160200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207717087588352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213266087262208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3878471217022 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38566100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214375416492660 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98217709 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213266104301568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213266087262208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203712892826624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42067666 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213266104301568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14185643 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214375519680606 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98217709 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56150043 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38565861 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42067668 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3890562061824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24380454 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98227868 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56150043 trainer/global_step‚5549 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14185407 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3878385068436 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330À•7[N2¦- _wandbruntime‚2204  _runtime‚2204³6ÄÁ§ý .loss/FRt2/assignment_loss‚1.14678062539798 1loss/SRqqt2/assignment_loss‚0.686557110968787 1loss/FBt1/assignment_loss‚0.17859963082661456 1loss/SRqqt1/detection_loss‚1.2561654558240556 $ _timestamp‚1.784499130833406e+09 _runtime‚2204.744079997 2loss/SRqqt1/assignment_loss‚0.8501787452996538 2loss/FBt2/assignment_loss‚0.055574570237035915 /loss/FRt2/detection_loss‚1.5668651352640197 %loss/total_loss‚11.81774691255466 trainer/global_step‚5549 0loss/FRt1/assižpZognment_loss‚2.7698348900701246 1loss/SRqqt2/detection_loss‚0.6476628531957649 .loss/FBt1/detection_loss‚1.215404492601837 /loss/FBt2/detection_loss‚0.8824492460489273 _step‚449 /loss/FRt1/detection_loss‚0.5616741568198612 epoch‚6£Ð?$¨‚ 0loss/FRt1/assignment_loss‚2.7698348900701246 1loss/SRqqt2/detection_loss‚0.6476628531957649 .loss/FBt1/detection_loss‚1.215404492601837 /loss/FBt2/detection_loss‚0.8824492460489273 _step‚449 /loss/FRt1/detection_loss‚0.5616741568198612 epoch‚6 .loss/FRt2/assignment_loss‚1.14678062539798 1loss/SRqqt2/assignment_loss‚0.686557110968787 1loss/FBt1/assignment_loss‚0.17859963082661456 1loss/SRqqt1/detection_loss‚1.2561654558240556 $ _timestamp‚1.784499130833406e+09 _runtime‚2204.744079997 2loss/SRqqt1/assignment_loss‚0.8501787452996538 2loss/FBt2/assignment_loss‚0.055574570237035915 /loss/FRt2/detection_loss‚1.5668651352640197 %loss/total_loss‚11.81774691255466 trainer/global_step‚5549Á0_Jš©‚(: »—õÒŒ», gpu.0.gpu‚70.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚191.594' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.86466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡)ÊçVª‚(:Ë »—õÒÞô¤-" proc.memory.rssMB‚ 2822.9609375( proc.memory.percent‚17.2299861907959 cpu‚26.29868810852869% disk.nvme1n1.out‚7577.64013671875 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚1341.64208984375 disk.nvme0c0n1.in‚ 0.02734375 proc.cpu.threads‚17' disk.nvme1c1n1.out‚7577.64013671875 disk.nvme1n1.in‚0# disk.dm-0.out‚17159.83447265625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11055.68408203125 network.sent‚ 7.328254e+06' disk./.usagePercent‚75.970986743176 disk.nvme0n1.in‚ 0.02734375 disk.nvme1c1n1.in‚0 disk.md0.out‚61.59375 disk.dm-0.in‚ 0.02734375 disk.rbd0.in‚0$ memory_percent‚99.19438362121582 network.recv‚670377% disk./.usageGB‚1166.3445739746094& disk.nvme0n1.out‚11055.68408203125' proc.memory.availableMB‚ 131.9921875<–'I«‚(:¾ Ê—õÒÉÇÊ+ disk.nvme0n1.in‚0.03125$ disk.nvme0n1.out‚11159.572265625 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2822.9609375) disk./.usagePercent‚75.98645847446657$ disk./.usageGB‚1166.582103729248 disk.md0.in‚0$ memory_percent‚99.19414520263672% proc.memory.availableMB‚ 132.03125( proc.memory.percent‚17.2299861907959 cpu‚26.296611935632406 proc.cpu.threads‚17 network.sent‚ 7.376509e+06 network.recv‚674404 disk.rbd0.out‚0& disk.nvme0c0n1.out‚11159.572265625' disk.nvme1c1n1.out‚7630.83349609375 disk.md0.out‚ 61.9453125! disk.dm-0.out‚17315.931640625 disk.nvme0c0n1.in‚0.03125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7630.83349609375 disk.dm-0.in‚0.03125 disk.dm-1.in‚0" disk.dm-1.out‚1341.86474609375 disk.rbd0.in‚0|5Àš¬‚(: Ê—õÒ’£‚, gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚163.741' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚54.58033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àv—2­- _wandbruntime‚2222  _runtime‚2222vÊÇ®Á trainer/global_step‚5599 % _timestamp‚1.7844991490185144e+09 _runtime‚2222.929476251 _step‚450 & lr-AdamW/pg1‚9.805126305537792e-05 & lr-AdamW/pg2‚9.805126305537792e-05È‹‹Ì¯Æ & lr-AdamW/pg1‚9.805126305537792e-05 & lr-AdamW/pg2‚9.805126305537792e-05 trainer/global_step‚5599 % _timestamp‚1.7844991490185144e+09 _runtime‚2222.929476251 _step‚450Â^dø2°- _wandbruntime‚2222  _runtime‚2222C©Ò'±ÌO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3923821693952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38897010 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212303779783824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216215301336864 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215097056295936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219020860950528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219020860950528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24590667 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215097039256576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99063292 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78449914901947e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56636897 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205461955531776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42426397 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42426395 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4038490200576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42426397 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42426395 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4038490200576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3923821693952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38897010 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212303779783824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216215301336864 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215097056295936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219020860950528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219020860950528û" 2³- _wandbruntime‚2223  _runtime‚2223§;¨ð´ØM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5599 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209537019450368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42433716 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3924499803648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212340634812224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4039368761344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216252849356068 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24594954 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99090670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3924587832832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216252952544014 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3912300692430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38904018 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24594957 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56646795 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215134405623808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4039190452224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219058905427456 _runtime‚2223.189280766 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14309061 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42433718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215134422663168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batcVÃmåîh_end/inactive_split.all.freed‚38903768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99090670 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42433716 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205497625785344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99080511 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209536816237568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3924499803648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3924587832832Yã&µÝM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209537019450368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42433716 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3924499803648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212340634812224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4039368761344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216252849356068 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24594954 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99090670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3924587832832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216252952544014 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3912300692430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38904018 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24594957 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56646795 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215134405623808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4039190452224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219058905427456 _runtime‚2223.189280766 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14309061 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42433718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215134422663168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38903768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99090670 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42433716 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205497625785344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99080511 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209536816237568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3924499803648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3924587832832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5599 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8078743815422058 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6136364936828613 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1317477971315384 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_1_of_4‚0.9677146673202515 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8928571343421936 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17424318194389343 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03315175324678421 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039579253643751144 *jet/accuracy_2_of_2‚0.1376599818468094 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18200257420539856 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7895989418029785 jet/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6454765200614929 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9781241416931152 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08808460086584091 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5990243554115295 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36637163162231445 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10074637085199356 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01289744209498167 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.982082188129425 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1566106528043747 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6785527467727661 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9641642570495605 particle/accuracy_2_of_3‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 jet/accuracy_1_of_5‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20737475156784058 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20173372328281403 +jet/accuracy_1_of_1‚0.29963213205337524 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8029913902282715 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799394011497498 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3656669557094574 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03350377082824707 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4960937798023224 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_3‚0.5358520746231079 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16972987353801727 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7081196308135986 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7736841440200806 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48247650265693665 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936230182647705 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2797320783138275 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978419542312622 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08196756988763809 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8479430079460144 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5443816184997559 +jet/accuracy_4_of_4‚0.01071428507566452 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5347222089767456 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962807893753052 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03481361269950867 *jet/accuracy_1_of_2‚0.7583601474761963 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2638888657093048 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8391938805580139 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6152624487876892 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.994744598865509 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5353829264640808 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025321239605545998 *jet/accuracy_2_of_4‚0.8114309906959534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9976457953453064 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *loss/val_total_loss‚10.238435032319877 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5602805018424988 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.798198938369751 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9871794581413269 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚2241.077295301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9930397868156433 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9976457953453064 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6075322031974792 +jet/accuracy_3_of_3‚0.05957479774951935 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15429095923900604 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962131977081299 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4974040985107422 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1436835080385208 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12078077346086502 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7497104406356812 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14779646694660187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7163678407669067 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9045501947402954 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6704764366149902 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.39652562141418457 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6453298330307007 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2142857164144516 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7599999904632568 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7890350222587585 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5536152124404907 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 trainer/global_step‚5606 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6745875477790833 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22627055644989014 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14732970297336578 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15000000596046448 particle/accuracy_5_of_5‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16962571442127228 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.811672568321228 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10781250149011612 +particle/sensitivity‚0.6442925930023193 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4410816431045532 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9868420958518982 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.613562822341919 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6227042078971863 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024947209283709526 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6222032904624939 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9952034950256348 +particle/specificity‚0.9036279320716858 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚454 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6823375225067139 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8448853492736816 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06511019915342331 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1440306007862091 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6967440247535706 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9918238520622253 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47276121377944946 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5154752135276794 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7304567694664001 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7357518076896667 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36600759625434875 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5992745161056519 particle/accuracy_2_of_4‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24186502397060394 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1774371862411499 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876543283462524 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9108504056930542 5validation_average_jet_accuracy‚0.344831258058548 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2142857164144516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9881266355514526 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006656848359853029 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12994444370269775 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01325961109250784 /particle/accuracy_2_of_2‚0.9998950958251953 particle/accuracy_3_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38571470975875854 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7385016083717346 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08328992128372192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9937435984611511 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6227042078971863 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645ù?ÏRº½Ù{ ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025321239605545998 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997228741645813 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3712121248245239 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 epoch‚6 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1503763496875763 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.773016631603241 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8576475977897644 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9916005730628967 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1566106528043747 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46654555201530457 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03557312488555908 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9829646348953247 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8078743815422058 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6136364936828613 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1317477971315384 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_1_of_4‚0.9677146673202515 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8928571343421936 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17424318194389343 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03315175324678421 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039579253643751144 *jet/accuracy_2_of_2‚0.1376599818468094 @&Purity/*FRt2SRqqt1FBt/evenßx¯%5t_proportion‚0.0009998140158131719 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18200257420539856 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7895989418029785 jet/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6454765200614929 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9781241416931152 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08808460086584091 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5990243554115295 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36637163162231445 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10074637085199356 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01289744209498167 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.982082188129425 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1566106528043747 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6785527467727661 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9641642570495605 particle/accuracy_2_of_3‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 jet/accuracy_1_of_5‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20737475156784058 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20173372328281403 +jet/accuracy_1_of_1‚0.29963213205337524 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8029913902282715 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799394011497498 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3656669557094574 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03350377082824707 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4960937798023224 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_3‚0.5358520746231079 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16972987353801727 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7081196308135986 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7736841440200806 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48247650265693665 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936230182647705 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2797320783138275 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978419542312622 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08196756988763809 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8479430079460144 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5443816184997559 +jet/accuracy_4_of_4‚0.01071428507566452 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5347222089767456 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962807893753052 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03481361269950867 *jet/accuracy_1_of_2‚0.7583601474761963 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2638888657093048 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8391938805580139 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6152624487876892 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.994744598865509 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5353829264640808 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025321239605545998 *jet/accuracy_2_of_4‚0.8114309906959534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9976457953453064 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *loss/val_total_loss‚10.238435032319877 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5602805018424988 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.798198938369751 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9871794581413269 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚2241.077295301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9930397868156433 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9976457953453064 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6075322031974792 +jet/accuracy_3_of_3‚0.05957479774951935 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15429095923900604 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962131977081299 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4974040985107422 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1436835080385208 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12078077346086502 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7497104406356812 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14779646694660187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7163678407669067 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9045501947402954 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6704764366149902 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.39652562141418457 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6453298330307007 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2142857164144516 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7599999904632568 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7890350222587585 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5536152124404907 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 trainer/global_step‚5606 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6745875477790833 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22627055644989014 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14732970297336578 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15000000596046448 particle/accuracy_5_of_5‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16962571442127228 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.811672568321228 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10781250149011612 +particle/sensitivity‚0.6442925930023193 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4410816431045532 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9868420958518982 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.613562822341919 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6227042078971863 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024947209283709526 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6222032904624939 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9952034950256348 +particle/specificity‚0.9036279320716858 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚454 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6823375225067139 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8448853492736816 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06511019915342331 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1440306007862091 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6967440247535706 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9918238520622253 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47276121377944946 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5154752135276794 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7304567694664001 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7357518076896667 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36600759625434875 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5992745161056519 particle/accuracy_2_of_4‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24186502397060394 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1774371862411499 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876543283462524 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9108504056930542 5validation_average_jet_accuracy‚0.344831258058548 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2142857164144516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9881266355514526 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006656848359853029 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12994444370269775 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01325961109250784 /particle/accuracy_2_of_2‚0.9998950958251953 particle/accuracy_3_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38571470975875854 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7385016083717346 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08328992128372192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9937435984611511 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6227042078971863 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9649845957756042 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4801676273345947 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6905872821807861 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.74094557762146 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16787388920783997 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919114112854004 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9077708125114441 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1300138533115387 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1286652833223343 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15000000596046448 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8936131596565247 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6147586107254028 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7672960162162781 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6075322031974792 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5307455658912659 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932742118835449 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8576475977897644 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1299366056919098 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7461776733398438 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03172961622476578 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7599999904632568 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6203956007957458 $ _timestamp‚1.784499167165964e+09Æãk H¾‚(:½ è—õÒ¿°è+& disk.nvme0n1.out‚11221.39892578125 disk.nvme1c1n1.in‚0" disk.nvme1c1n1.out‚ 7749.390625 disk.md0.out‚ 62.5078125( disk.nvme0c0n1.out‚11221.39892578125 proc.cpu.threads‚17 network.sent‚ 7.456656e+06 network.recv‚682376) disk./.usagePercent‚75.98647263748525% disk./.usageGB‚1166.5823211669922 disk.nvme0n1.in‚0.03125 disk.nvme1n1.out‚ 7749.390625( proc.memory.availableMB‚ 191.63671875 cpu‚26.313331788072603 disk.dm-0.in‚0.03125 disk.dm-1.in‚0 disk.dm-1.out‚1405.939453125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.03125$ memory_percent‚98.83034229278564) proc.memory.percent‚17.29283332824707# disk.dm-0.out‚17431.02587890625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2833.2578125«<.Ž›¿‚(: è—õÒ³þ, gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚179.335' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚59.778333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ásÿø2À- _wandbruntime‚2257  _runtime‚2257%¤M~ÇÁÁ & lr-AdamW/pg1‚9.797459325724146e-05 & lr-AdamW/pg2‚9.797459325724146e-05 trainer/global_step‚5649 % _timestamp‚1.7844991837057734e+09 _runtime‚2257.618386701 _step‚455çM+ºÌÂÆ _runtime‚2257.618386701 _step‚455 & lr-AdamW/pg1‚9.797459325724146e-05 & lr-AdamW/pg2‚9.797459325724146e-05 trainer/global_step‚5649 % _timestamp‚1.7844991837057734e+09Ç•f²2Ã- _wandbruntime‚2257  _runtime‚2257õ°;ïØ'ÄÒO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217938021213184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217938021213184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43029915 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219081617676190 _step‚456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100340973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57311058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14508175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221916200966144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215115872640176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3978179752960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221916306027520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3978267774976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39407673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39407419 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57300915 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100330828 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219081720864024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3978179752960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57300915 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221916306027520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4093969889280 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43029913 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217938038252544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4094152399872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2257.618453151 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217938038252544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24899244 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221916200966144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24899247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57311058 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215115855600816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3965848223848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208172111754240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43029913 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272CþÄöêÅ×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57311058 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215115855600816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3965848223848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208172111754240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43029913 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217938021213184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217938021213184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43029915 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219081617676190 _step‚456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100340973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57311058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14508175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221916200966144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215115872640176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3978179752960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221916306027520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3978267774976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39407673 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39407419 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57300915 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100330828 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219081720864024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3978179752960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57300915 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221916306027520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4093969889280 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43029913 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217938038252544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4094152399872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2257.618453151 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217938038252544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24899244 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221916200966144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24899247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39414174 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208207831815168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39414433 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7844991839619029e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100358206 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4094852541952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100358206 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217975404619776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57310813 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217975404619776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208207806911488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚5649 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3978857862656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221954245443072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217975387580416 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966524175404 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215152727668576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43037236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221954350511616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219119268883340 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100348047 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217975387580416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3978945891840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152íî( ç&ÈáM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _step‚457 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43037236 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221954350511616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4094852541952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100358206 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217975404619776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57310813 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217975404619776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208207806911488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚5649 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3978857862656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221954245443072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217975387580416 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966524175404 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215152727668576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43037236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221954350511616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219119268883340 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100348047 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217975387580416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3978945891840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57320970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14510640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212302684357120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39414174 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208207831815168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39414433 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7844991839619029e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100358206 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4012205516800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39745883 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43395965 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219806339574784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223818457069568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4129116729344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43395963 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57797667 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219806356614144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4012117494784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101203775 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4128936315904 _step‚460 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101193630 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14632388 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4012205516800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39745883 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43395965 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219806339574784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223818457069568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4129116729344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43395963 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57797667 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219806356614144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4012117494784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101203775 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4128936315904 _step‚460 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101193630 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14632388 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209992589677568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101221008 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223856606615040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14634887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101221008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214122195494400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209992564773888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57817722 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216996348095936 _runtime‚2276.786885322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223856501546496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223856606615040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43403284 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219843705942016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39752924 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129821612544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57817722 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57807565 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4012795604480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4000353650812 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39752655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101210849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43403286 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43403286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25118034 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214122411290112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43403284 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219843705942016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216996331056576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚461 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209992589677568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101221008 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223856606615040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14634887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101221008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214122195494400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209992564773888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57817722 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216996348095936 _runtime‚2276.786885322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223856501546496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223856606615040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43403284 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219843705942016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39752924 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129821612544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57817722 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57807565 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4012795604480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4000353650812 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39752655 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101210849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43403286 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43403286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2511803Pwê[C4 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214122411290112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43403284 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219843705942016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216996331056576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚461 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222836483403006 _runtime‚2296.255042086 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225720818234368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221674657936384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211741627479040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4046055236608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225720713172992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14756350 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218803079416176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43762015 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25328247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚5749 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215905497166848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215905706678272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40084597 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4033507174664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221674674975744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58304562 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4033421026190 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4046055236608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102066577ëG’[Ý'ã×O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40084597 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4033507174664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221674674975744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58304562 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4033421026190 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4046055236608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102066577 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211741652382720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14756106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58294419 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40084350 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844992223439953e+09 _step‚464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225720713172992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43762015 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222836483403006 _runtime‚2296.255042086 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225720818234368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221674657936384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211741627479040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4046055236608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225720713172992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14756350 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218803079416176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43762015 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25328247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚5749 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215905497166848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215905706678272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928ÐÄ2ÜË2ä- _wandbruntime‚2296  _runtime‚2296?tâmåÜM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14758821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14758577 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚5749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4046821375488 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58314474 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4034096977634 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102083810 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218839934444576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102073651 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225758757649920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221712041342976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102073651 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4164748768768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211777347539968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚465 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43769336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚2296.524449299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43769334 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43769336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25332534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102083810 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚225758862718464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221712041342976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222874031422210 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58314474 >8DeviceStatsMonitor.on_train_batch_end/Ununum_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43769334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58304317 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225758862718464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211777322636288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4164566265344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40091358 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4046733346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215941888901632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218839951483936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4046821375488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40091111 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215942096308736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222874134610156 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330> Á ç&æáM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40091358 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4046733346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215941888901632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218839951483936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4046821375488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40091111 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215942096308736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222874134610156 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14758821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14758577 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚5749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4046821375488 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58314474 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4034096977634 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102083810 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218839934444576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102073651 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225758757649920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221712041342976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102073651 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4164748768768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211777347539968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚465 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43769336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚2296.524449299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43769334 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43769336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25332534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102083810 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚225758862718464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221712041342976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222874031422210 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58314474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43769334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58304317 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225758862718464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211777322636288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4164566265344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482Ñ°EE2ç- _wandbruntime‚2296  _runtime‚2296þHU•è€ 0loss/FRt1/assignment_loss‚2.5574858809898053 /loss/FRt2/assignment_loss‚0.951844004519058 1loss/FBt1/assignment_loss‚0.22011686769689312 /loss/FRt2/detection_loss‚1.5073903633800165 1loss/SRqqt2/detection_loss‚0.6627608475175997 epoch‚7 /loss/FRt1/detection_loss‚0.6448701684648586 0loss/SRqqt1/detection_loss‚1.089965799092492 /loss/FBt1/detection_loss‚0.7438400308960207 &loss/total_loss‚10.654356361204218 _runtime‚2296.524970779 2loss/SRqqt2/assignment_loss‚0.6142766332246828 /loss/FBt2/detection_loss‚0.6908254144010973 trainer/global_step‚5749 % _timestamp‚1.7844992226143382e+09 _step‚466 2loss/SRqqt1/assignment_loss‚0.9416424512577278 1loss/FBt2/assignment_loss‚0.02933789976396674“ é… 0loss/FRt1/assignment_loss‚2.5574858809898053 /loss/FRt2/assignment_loss‚0.951844004519058 1loss/FBt1/assignment_loss‚0.22011686769689312 /loss/FRt2/detection_loss‚1.5073903633800165 1loss/SRqqt2/detection_loss‚0.6627608475175997 epoch‚7 /loss/FRt1/detection_loss‚0.6448701684648586 0loss/SRqqt1/detection_loss‚1.089965799092492 /loss/FBt1/detection_loss‚0.7438400308960207 &loss/total_loss‚10.654356361204218 _runtime‚2296.524970779 2loss/SRqqt2/assignment_loss‚0.6142766332246828 /loss/FBt2/detection_loss‚0.6908254144010973 trainer/global_step‚5749 % _timestamp‚1.7844992226143382e+09 _step‚466 2loss/SRqqt1/assignment_loss‚0.9416424512577278 1loss/FBt2/assignment_loss‚0.02933789976396674Ò5YbAê‚(:¶ ¤˜õÒ¨£¥+ disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚11247.62109375) proc.memory.percent‚17.29283332824707 network.recv‚698400) disk./.usagePercent‚75.99052500014312 disk.dm-1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 63.91796875( proc.memory.availableMB‚ 126.78515625% disk./.usageGB‚1166.6445350646973 disk.nvme0c0n1.in‚0.03125% disk.nvme0c0n1.out‚11247.62109375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚7871.126953125# disk.nvme1n1.out‚7871.126953125 disk.dm-0.in‚0.03125 disk.dm-1.out‚1470.490234375 disk.nvme0n1.in‚0.03125$ memory_percent‚99.22616481781006" proc.memory.rssMB‚ 2833.2578125 cpu‚26.314291305050496 proc.cpu.threads‚17 network.sent‚ 7.629334e+06 disk.dm-0.out‚ 17511.375Š+›ë‚(: ¤˜õÒÔýü+ gpu.0.gpu‚72.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚113.681' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.89366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÒú‘2ì- _wandbruntime‚2316  _runtime‚2316ó¸­áÇíÁ & lr-AdamW/pg1‚9.774006281872046e-05 & lr-AdamW/pg2‚9.774006281872046e-05 trainer/global_step‚5799 % _timestamp‚1.7844992426298869e+09 _runtime‚2316.540845104 _step‚467fävdÌîÆ trainer/global_step‚5799 % _timestamp‚1.7844992426298869e+09 _runtime‚2316.540845104 _step‚467 & lr-AdamW/pg1‚9.774006281872046e-05 & lr-AdamW/pg2‚9.774006281872046e-05Ó'¥cÛ2ï- _wandbruntime‚2316  _runtime‚2316¹¨v?Ø'ðÒO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44128063 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213526385341440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844992426309001e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58791171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223542976297984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223542993337344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58791171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4067336650072 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217725150773760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _step‚468 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25542744 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44128065 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102929379 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40422841 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44128065 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14880094 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220646665764816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44128063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224713916266414 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227623074337792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58801314 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25542747 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227622969276416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224714019454248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102919234 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4080081000448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4079992978432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40422566 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217725362382336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213526410245120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4080081000448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14879822 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4198952137216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4198765432320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223542976297984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223542993337344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58801314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220646682804176 _runtime‚2316.545420861 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4079992978432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227622969276416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Ü ·[[ñ×O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58791171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223542976297984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223542993337344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58791171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4067336650072 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217725150773760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _step‚468 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25542744 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44128065 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102929379 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40422841 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44128065 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14880094 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220646665764816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44128063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224713916266414 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227623074337792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58801314 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25542747 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227622969276416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224714019454248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102919234 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4080081000448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4079992978432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40422566 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚2177253623829^[‚336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213526410245120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4080081000448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14879822 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4198952137216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4198765432320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223542976297984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223542993337344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58801314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220646682804176 _runtime‚2316.545420861 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4079992978432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227622969276416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44128063 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213526385341440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7844992426309001e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712ÔØ=2ò- _wandbruntime‚2316  _runtime‚2316¹ÓKâ&óÜM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4199649395200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2316.812518718 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58801069 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58811226 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4080671088128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚5799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227661013753344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227661118821888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58801069 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58811226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4067926453042 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40429267 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217761754797568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102946612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25547034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7844992429014053e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102946612 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4080759117312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223580359704576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102936453 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25547037 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220683537832576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44135386 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44135384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40429517 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220683554871936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44135386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223580342665216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223580359704576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14882480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4068012601628 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4080759117312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4080671088128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217761549487616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7844992429014053e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102946612 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4080759117312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223580359704576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102936453 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25547037 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220683537832576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44135386 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44135384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40429517 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220683554871936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44135386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223580342665216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223580359704576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14882480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4068012601628 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4080759117312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4080671088128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217761549487616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2316.812518718 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58801069 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58811226 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4080671088128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚5799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227661013753344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227661118821888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58801069 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58811226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4067926453042 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40429267 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217761754797568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102946612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25547034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E'¥àöÿ /loss/FRt1/detection_loss‚0.6197867745950288 $ _timestamp‚1.784499242902444e+09 .loss/FRt1/assignment_loss‚2.56646192566318 2loss/SRqqt1/assignment_loss‚0.7318200296824322 1loss/FBt1/assignment_loss‚0.16290811375724465 0loss/SRqqt2/detection_loss‚0.677177433702303 /loss/FBt1/detection_loss‚0.8386379840834707 trainer/global_step‚5799 0loss/FRt2/assignment_loss‚1.1404086011814407 2loss/SRqqt2/assignment_loss‚0.6254883092404305 2loss/FBt2/assignment_loss‚0.021077130400897905 /loss/FRt2/detection_loss‚1.4796407145273638 1loss/SRqqt1/detection_loss‚1.1826211271814349 epoch‚7 _step‚470 /loss/FBt2/detection_loss‚0.7059590288748344 &loss/total_loss‚10.751987172890061 _runtime‚2316.816370575Ó™ ÷„ _runtime‚2316.816370575 /loss/FRt1/detection_loss‚0.6197867745950288 $ _timestamp‚1.784499242902444e+09 .loss/FRt1/assignment_loss‚2.56646192566318 2loss/SRqqt1/assignment_loss‚0.7318200296824322 1loss/FBt1/assignment_loss‚0.16290811375724465 0loss/SRqqt2/detection_loss‚0.677177433702303 /loss/FBt1/detection_loss‚0.8386379840834707 trainer/global_step‚5799 0loss/FRt2/assignment_loss‚1.1404086011814407 2loss/SRqqt2/assignment_loss‚0.6254883092404305 2loss/FBt2/assignment_loss‚0.021077130400897905 /loss/FRt2/detection_loss‚1.4796407145273638 1loss/SRqqt1/detection_loss‚1.1826211271814349 epoch‚7 _step‚470 /loss/FBt2/detection_loss‚0.7059590288748344 &loss/total_loss‚10.751987172890061Ö‚2?²?ø‚(:´ ³˜õÒ¢ÌÏ+) disk./.usagePercent‚75.99052301235103# disk.nvme1n1.out‚7871.603515625 disk.md0.in‚0 disk.dm-1.in‚0$ memory_percent‚99.22676086425781 disk.md0.out‚ 64.1796875 disk.dm-1.out‚1470.583984375 disk.nvme0n1.in‚0.03125# disk.nvme0n1.out‚11248.38671875$ proc.memory.availableMB‚126.6875 proc.cpu.threads‚17 network.sent‚ 7.705521e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚0.03125 disk.dm-0.out‚ 17511.921875) proc.memory.percent‚17.29283332824707 network.recv‚702731% disk./.usageGB‚1166.6445045471191 disk.nvme0c0n1.in‚0.03125% disk.nvme0c0n1.out‚11248.38671875 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚7871.603515625" proc.memory.rssMB‚ 2833.2578125 cpu‚26.319751858994685 disk.rbd0.in‚0 disk.rbd0.out‚0}©È™ù‚(:Ž ³˜õÒ—ñû+ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.38' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.126666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ƒ…J2ú- _wandbruntime‚2336  _runtime‚2336eíÝgÇûÁ & lr-AdamW/pg1‚9.766038316504626e-05 & lr-AdamW/pg2‚9.766038316504626e-05 trainer/global_step‚5849 % _timestamp‚1.7844992626197855e+09 _runtime‚2336.530684061 _step‚471–¦"©ÌüÆ _runtime‚2336.530684061 _step‚471 & lr-AdamW/pg1‚9.766038316504626e-05 & lr-AdamW/pg2‚9.766038316504626e-05 trainer/global_step‚5849 % _timestamp‚1.7844992626197855e+09×ÂBïP2ý- _wandbruntime‚2336  _runtime‚2336¡p»£þÒO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229525330441216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40761884 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222490269152816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44494113 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40761629 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215311168107520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229525225379840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44494115 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59287923 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15004385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219544887821312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4233922933760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226591349129822 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4113930720256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229525330441216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44494115 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7844992626207085e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103782036 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114018742272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225411294659584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4114018742272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225411311698944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226591452317656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4101079977006 _step‚472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103792181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59298066 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4101166125480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25757244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4113930720256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222490286192176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229525225379840 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225411311698944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226591452317656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4101079977006 _step‚472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103792181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59298066 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4101166125480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25757244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4113930720256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222490286192176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229525225379840 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229525225379840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44494115 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59287923 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15004385 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219544887821312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4233922933760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226591349129822 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4113930720256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229525330441216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44494115 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7844992626207085e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103782036 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114018742272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376ØÕš¬2€- _wandbruntime‚2336  _runtime‚2336æ/ªâ&ÜM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103799255 _step‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103809414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15006809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219581492251136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25761537 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222527141220576 trainer/global_step‚5849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59297821 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40768343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225448678066176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219581276455424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4234628986368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225448678066176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215346838361088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59307978 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44501436 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15007067 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40768604 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229563269856768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229563374925312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226628897149026 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4101755928450 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4234438094336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4101842077036 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44501436 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225448661026816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4114608829952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103809414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44501434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44501434 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59297821 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4114696859136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4114696859136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2336.800590576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229563374925312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7844992628895476e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225448661026816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59307978 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103799255 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1HÎ%‚áM _runtime‚2336.800590576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229563374925312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7844992628895476e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225448661026816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59307978 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103799255 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103799255 _step‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103809414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15006809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219581492251136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25761537 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222527141220576 trainer/global_step‚5849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59297821 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40768343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225448678066176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219581276455424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4234628986368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225448678066176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215346838361088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59307978 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44501436 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15007067 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40768604 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚229563269856768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229563374925312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226628897149026 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4101755928450 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.cur‡T©¡Â rent‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4234438094336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4101842077036 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44501436 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225448661026816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4114608829952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103809414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44501434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44501434 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59297821 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4114696859136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4114696859136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ù¡í5_2ƒ- _wandbruntime‚2336  _runtime‚2336ÜÍ„ÿ 2loss/SRqqt2/assignment_loss‚0.7922311881081154 .loss/FBt1/assignment_loss‚0.31357883120084 1loss/SRqqt1/detection_loss‚1.0966270184317917 epoch‚7 2loss/SRqqt1/assignment_loss‚0.8342125079368953 2loss/FBt2/assignment_loss‚0.007621540053010407 1loss/SRqqt2/detection_loss‚0.6606333105868624 /loss/FBt2/detection_loss‚0.7643927301702299 _runtime‚2336.801121877 /loss/FRt1/assignment_loss‚2.437999195989533 /loss/FRt1/detection_loss‚0.6166530420981519 &loss/total_loss‚10.893476699374009 % _timestamp‚1.7844992628905096e+09 _step‚474 0loss/FRt2/assignment_loss‚1.0472967155436252 /loss/FRt2/detection_loss‚1.3423254731778347 /loss/FBt1/detection_loss‚0.9799051460771201 trainer/global_step‚5849rºz‹ …„ 2loss/SRqqt2/assignment_loss‚0.7922311881081154 .loss/FBt1/assignment_loss‚0.31357883120084 1loss/SRqqt1/detection_loss‚1.0966270184317917 epoch‚7 2loss/SRqqt1/assignment_loss‚0.8342125079368953 2loss/FBt2/assignment_loss‚0.007621540053010407 1loss/SRqqt2/detection_loss‚0.6606333105868624 /loss/FBt2/detection_loss‚0.7643927301702299 _runtime‚2336.801121877 /loss/FRt1/assignment_loss‚2.437999195989533 /loss/FRt1/detection_loss‚0.6166530420981519 &loss/total_loss‚10.893476699374009 % _timestamp‚1.7844992628905096e+09 _step‚474 0loss/FRt2/assignment_loss‚1.0472967155436252 /loss/FRt2/detection_loss‚1.3423254731778347 /loss/FBt1/detection_loss‚0.9799051460771201 trainer/global_step‚5849Ú7"€ÎN†‚(:à ˜õÒŠÕâ+ disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.03515625" proc.memory.rssMB‚ 2833.2578125) proc.memory.percent‚17.29283332824707 proc.cpu.threads‚17 disk.md0.out‚ 64.484375" disk.nvme0n1.out‚ 11249.4609375$ memory_percent‚99.22676086425781 network.sent‚ 7.766799e+06( disk./.usagePercent‚75.9902163954205! disk.dm-1.out‚1470.7216796875 disk.nvme0c0n1.in‚ 0.03515625$ disk.nvme0c0n1.out‚ 11249.4609375$ proc.memory.availableMB‚126.6875 cpu‚26.324403396476352 network.recv‚706853% disk./.usageGB‚1166.6397972106934' disk.nvme1c1n1.out‚7873.91064453125 disk.md0.in‚0 disk.dm-0.in‚ 0.03515625# disk.dm-0.out‚17514.49072265625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7873.91064453125.on›‡‚(: ˜õÒåߊ, gpu.0.gpu‚83.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.556' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.51866666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Á$Pˆ‚(:Å јõÒëê·+) proc.memory.percent‚17.29283332824707$ disk.nvme1c1n1.out‚ 7877.51171875" disk.nvme1n1.out‚ 7877.51171875 disk.rbd0.out‚0$ proc.memory.availableMB‚126.6875 cpu‚26.321934074522837 network.sent‚ 7.816598e+06 network.recv‚711746 disk.dm-0.in‚ 0.03515625# disk.dm-0.out‚17515.55126953125 proc.cpu.threads‚17) disk./.usagePercent‚75.99024670924994% disk./.usageGB‚1166.6402626037598( disk.nvme0c0n1.out‚11250.96240234375 disk.nvme0n1.in‚ 0.03515625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚64.96875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.03515625& disk.nvme0n1.out‚11250.96240234375 disk.nvme1c1n1.in‚0! disk.dm-1.out‚1473.7177734375 disk.rbd0.in‚0$ memory_percent‚99.22676086425781" proc.memory.rssMB‚ 2833.2578125ê–UEš‰‚(: јõÒËÜò+ gpu.0.gpu‚78.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.444' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.48133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì íÑ2Š- _wandbruntime‚2356  _runtime‚2356Ñ_hÇ‹Á _step‚475 & lr-AdamW/pg1‚9.757995420798569e-05 & lr-AdamW/pg2‚9.757995420798569e-05 trainer/global_step‚5899 % _timestamp‚1.7844992826130888e+09 _runtime‚2356.524019305ÞÚ ÌŒÆ & lr-AdamW/pg1‚9.757995420798569e-05 & lr-AdamW/pg2‚9.757995420798569e-05 trainer/global_step‚5899 % _timestamp‚1.7844992826130888e+09 _runtime‚2356.524019305 _step‚475Û̇Ë2- _wandbruntime‚2356  _runtime‚2356iFWØ'ŽÒO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104654983 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221364761102848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224333872540816 _runtime‚2356.528410892 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59794818 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228468781993230 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4134995600888 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104644838 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4147868462080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4147956484096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15128443 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217095901066240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚476 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59794818 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231427586544640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231427481483264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228468885181064 % _timestamp‚1.7844992826140106e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221364545299968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217095925969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4147956484096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59784675 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231427481483264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41100190 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44860163 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4147868462080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25971747 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4268644233728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44860165 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚476 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59794818 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231427586544640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231427481483264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228468885181064 % _timestamp‚1.7844992826140106e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221364545299968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217095925969920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4147956484096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59784675 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231427481483264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41100190 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44860163 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4147868462080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25971747 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4268644233728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44860165 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚2356.795443291 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44867484 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224370761647936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227316996427776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221400932344832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5899 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104672216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4135585403858 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44867486 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4269336121344 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227316996427776 REDeviceStatsMonitor.on_train]M_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4148634600960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59804730 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227316979388416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15130691 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4269531207680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4148546571776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41106984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221401152334848 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41106725 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104662057 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25976034 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44867486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231465631028736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104662057 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217131596223488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44867484 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59794573 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59794573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227316979388416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224370744608576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104672216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59804730 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4148634600960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚477 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4148546571776E#oÒæ&’àM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227316996427776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4148634600960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59804730 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227316979388416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15130691 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4269531207680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4148546571776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41106984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221401152334848 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41106725 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104662057 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25976034 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44867486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231465631028736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104662057 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217131596223488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44867484 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59794573 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59794573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227316979388416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224370744608576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104672216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59804730 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4148634600960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚477 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4148546571776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231465525960192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15130947 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚2356.795443291 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44867484 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224370761647936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227316996427776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221400932344832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5899 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104672216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4135585403858 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44867486 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4269336121344 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ýœ“íg2“- _wandbruntime‚2356  _runtime‚2356K”k”€ /loss/FRt2/assignment_loss‚1.094720230112231 1loss/SRqqt1/detection_loss‚1.1724461805271476 epoch‚7 trainer/global_step‚5899 _step‚478 &loss/total_loss‚11.608206437456225 2loss/SRqqt1/assignment_loss‚0.7265225250520778 0loss/FBt1/assignment_loss‚0.1530327431648532 /loss/FRt2/detection_loss‚1.5987435234180554 /loss/FBt1/detection_loss‚1.4959406895637974 /loss/FBt2/detection_loss‚0.7965338372321695 % _timestamp‚1.7844992828853834e+09 2loss/SRqqt2/assignment_loss‚0.6916193868657241 /loss/FRt1/detection_loss‚0.6344116077527284 _runtime‚2356.799382627 0loss/FRt1/assignment_loss‚2.4694936287379314 1loss/FBt2/assignment_loss‚0.08688290073390487 1loss/SRqqt2/detection_loss‚0.6878591842956042ÊŸJ •… trainer/global_step‚5899 _step‚478 &loss/total_loss‚11.608206437456225 2loss/SRqqt1/assignment_loss‚0.7265225250520778 0loss/FBt1/assignment_loss‚0.1530327431648532 /loss/FRt2/detection_loss‚1.5987435234180554 /loss/FBt1/detection_loss‚1.4959406895637974 /loss/FBt2/detection_loss‚0.7965338372321695 % _timestamp‚1.7844992828853834e+09 2loss/SRqqt2/assignment_loss‚0.6916193868657241 /loss/FRt1/detection_loss‚0.6344116077527284 _runtime‚2356.799382627 0loss/FRt1/assignment_loss‚2.4694936287379314 1loss/FBt2/assignment_loss‚0.08688290073390487 1loss/SRqqt2/detection_loss‚0.6878591842956042 /loss/FRt2/assignment_loss‚1.094720230112231 1loss/SRqqt1/detection_loss‚1.1724461805271476 epoch‚7ÞEÁõW–‚(:Ì à˜õÒ¹Ö€, disk.nvme0n1.in‚ 0.03515625 disk.dm-0.in‚ 0.03515625 proc.cpu.threads‚17% disk./.usageGB‚1166.6401824951172$ disk.nvme1n1.out‚7878.5146484375 disk.dm-1.out‚1473.947265625( proc.memory.availableMB‚ 126.65234375" proc.memory.rssMB‚ 2833.2578125) proc.memory.percent‚17.29283332824707 cpu‚26.31989457049355( disk.nvme0c0n1.out‚11253.62939453125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚7878.5146484375 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚17517.93701171875 disk.dm-1.in‚0 network.sent‚ 7.835304e+06) disk./.usagePercent‚75.99024149129569 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.03515625& disk.nvme0n1.out‚11253.62939453125 disk.md0.out‚ 65.45703125" memory_percent‚99.226975440979 network.recv‚714936à&tŠœ—‚(:‘ à˜õÒî–­, gpu.0.gpu‚90.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚190.723' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.574333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 êaZ2˜- _wandbruntime‚2376  _runtime‚2376Ÿñ¸Ç™Á & lr-AdamW/pg2‚9.749877723577174e-05 trainer/global_step‚5949 % _timestamp‚1.7844993023404279e+09 _runtime‚2376.251379463 _step‚479 & lr-AdamW/pg1‚9.749877723577174e-05 2zÌšÆ & lr-AdamW/pg1‚9.749877723577174e-05 & lr-AdamW/pg2‚9.749877723577174e-05 trainer/global_step‚5949 % _timestamp‚1.7844993023404279e+09 _runtime‚2376.251379463 _step‚479ß}ø2›- _wandbruntime‚2376  _runtime‚2376¥õß ×'œÑO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233329842648064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4168825076296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226177475928816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229147948422144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229147931382784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4181894225920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218880658928640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26186247 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41437778 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60291570 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230346318044472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218880683832320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226177492968176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26186244 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60281427 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233329737586688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45226215 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60281427 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223184188622336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4181894225920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15251534 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4181806203904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4303529693696 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844993023413618e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105507640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60291570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚233329842648064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45226213 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 _step‚480ÅFá¶ÖO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26186244 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60281427 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233329737586688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45226215 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60281427 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223184188622336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4181894225920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15251534 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4181806203904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4303529693696 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7844993023413618e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105507640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60291570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚233329842648064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45226213 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 _step‚480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233329842648064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4168825076296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226177475928816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229147948422144 RDDevice=·&StatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229147931382784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4181894225920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218880658928640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26186247 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41437778 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60291570 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230346318044472 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218880683832320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226177492968176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474àáSÏ2ž- _wandbruntime‚2376  _runtime‚2376‘{ðá&ŸÛM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26190537 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41444567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15254033 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218916354085888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41444822 trainer/global_step‚5949 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4182484313600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚481 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233367782063616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230383762875842 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4169414879266 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45233536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚233367887132160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105524859 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229185314789376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229185297750016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105535018 _runtime‚2376.506422229 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105535018 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60291325 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4304234270720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60301482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60291325 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229185314789376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105524859 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4169501027852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4304412579840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233367887132160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230383866063788 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45233534 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226214347996576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60301482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚07Q.æ& àM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4182484313600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229185297750016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41444567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15254033 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218916354085888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41444822 trainer/global_step‚5949 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4182484313600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚481 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233367782063616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230383762875842 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4169414879266 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45233536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚233367887132160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105524859 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229185314789376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229185297750016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105535018 _runtime‚2376.506422229 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105535018 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60291325 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4304234270720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60301482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60291325 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229185314789376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105524859 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4169501027852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4304412579840 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233367887132160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230383866063788 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45233534 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226214347996576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60301482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26190537 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26400744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106380587 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231016266783744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228021096356176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41775339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4215831967744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15374595 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106370442 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232223750907880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚5999 _runtime‚2395.672375567 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60778179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232223647720046 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235232098751488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4338621225472 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45592265 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231016249744384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215743945728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60788322 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚225004062920192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26400747 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7844993217613986e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4215831967744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45592263 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235231993690112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330É•KÉÝ'«×O eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220665441694720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4338442909184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228021079316816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4202568403230 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60778179 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231016266783744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106380587 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231016266783744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228021096356176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41775339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4215831967744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15374595 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106370442 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232223750907880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚5999 _runtime‚2395.672375567 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60778179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232223647720046 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235232098751488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4338621225472 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45592265 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231016249744384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215743945728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60788322 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚225004062920192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26400747 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7844993217613986e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4215831967744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45592263 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235231993690112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235231993690112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45592265 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235232098751488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚225003859700224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26400744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220701111948288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4216422055424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232261298927196 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235270143235584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339324111872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4339143705600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231053633150976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106397820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106387661 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26405034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚228057968423936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231053616111616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15377020 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 % _timestamp‚1.7844993220185533e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41782054 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232261195739250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4203244354674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚485 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45599584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26405037 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15377276 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚225040255653888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106387661 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208IL{o®áM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220701111948288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4216422055424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232261298927196 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235270143235584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339324111872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4339143705600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231053633150976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106397820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106387661 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26405034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚228057968423936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231053616111616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15377020 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 % _timestamp‚1.7844993220185533e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41782054 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232261195739250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4203244354674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚485 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45599584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26405037 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15377276 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚225040255653888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106387661 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60798234 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚228057951384576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106397820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41782313 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482åztVI2¯- _wandbruntime‚2395  _runtime‚2395WÈ’°þ 0loss/FRt1/assignment_loss‚2.3512810229735694 1loss/FBt2/assignment_loss‚0.05018652676711868 /loss/FRt2/detection_loss‚1.4999776411706245 0loss/SRqqt1/detection_loss‚1.086067291677595 .loss/FBt1/detection_loss‚1.253829276189208 0loss/FRt2/assignment_loss‚1.0665277392845454 2loss/SRqqt2/assignment_loss‚0.6412594769825358 /loss/FRt1/detection_loss‚0.5669176825032758 /loss/FBt2/detection_loss‚0.6035269257145759 epoch‚7 trainer/global_step‚5999 % _timestamp‚1.7844993220194619e+09 _runtime‚ 2395.9301105 1loss/FBt1/assignment_loss‚0.07737390750483505 2loss/SRqqt1/assignment_loss‚0.6623934789127867 1loss/SRqqt2/detection_loss‚0.6936757410250354 &loss/total_loss‚10.553016710705705 _step‚486Q½Ðr ±ƒ /loss/FRt1/detection_loss‚0.5669176825032758 /loss/FBt2/detection_loss‚0.6035269257145759 epoch‚7 trainer/global_step‚5999 % _timestamp‚1.7844993220194619e+09 _runtime‚ 2395.9301105 1loss/FBt1/assignment_loss‚0.07737390750483505 2loss/SRqqt1/assignment_loss‚0.6623934789127867 1loss/SRqqt2/detection_loss‚0.6936757410250354 &loss/total_loss‚10.553016710705705 _step‚486 0loss/FRt1/assignment_loss‚2.3512810229735694 1loss/FBt2/assignment_loss‚0.05018652676711868 /loss/FRt2/detection_loss‚1.4999776411706245 0loss/SRqqt1/detection_loss‚1.086067291677595 .loss/FBt1/detection_loss‚1.253829276189208 0loss/FRt2/assignment_loss‚1.0665277392845454 2loss/SRqqt2/assignment_loss‚0.6412594769825358愵ïž^²‚(:Ó þ˜õÒ¯éÌ+ disk.dm-0.in‚ 0.03515625# disk.dm-0.out‚17524.38623046875 disk.dm-1.in‚0" disk.dm-1.out‚1474.26220703125 disk.rbd0.in‚0' disk.nvme1c1n1.out‚7880.20166015625 disk.md0.in‚0% disk./.usageGB‚1166.6449356079102( disk.nvme0c0n1.out‚11260.52099609375 disk.rbd0.out‚0$ memory_percent‚99.22778606414795( proc.memory.availableMB‚ 126.51953125" proc.memory.rssMB‚ 2833.2578125) proc.memory.percent‚17.29283332824707 cpu‚26.330327386448076 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.md0.out‚ 66.30859375 network.sent‚ 7.959132e+06 network.recv‚724138) disk./.usagePercent‚75.99055108991436 disk.nvme0c0n1.in‚ 0.03515625 disk.nvme0n1.in‚ 0.03515625& disk.nvme0n1.out‚11260.52099609375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7880.20166015625eB’åš³‚(: þ˜õÒàé÷+ gpu.0.gpu‚57.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚194.899' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.96633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¥2´- _wandbruntime‚2414  _runtime‚2414ì\ç¨ÆµÀ _step‚487 & lr-AdamW/pg1‚9.733418445869974e-05 & lr-AdamW/pg2‚9.733418445869974e-05 trainer/global_step‚6049 $ _timestamp‚1.784499340671268e+09 _runtime‚2414.583380704@驫˶Š_runtime‚2414.583380704 _step‚487 & lr-AdamW/pg1‚9.733418445869974e-05 & lr-AdamW/pg2‚9.733418445869974e-05 trainer/global_step‚6049 $ _timestamp‚1.784499340671268e+09çreÓï2·- _wandbruntime‚2414  _runtime‚2414À  Ø'¸ÒO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226823747677184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45958313 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚237134354854912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229864699744176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61285074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844993406718915e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107243389 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4373371900928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107233244 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232884568105984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42113356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15498375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958313 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222450174653440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222450199557120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚2414.583445554 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232884585145344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4249681687552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237134249793536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚6049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958315 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4236397878638 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61285074 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229864682704816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237134249793536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61274931 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4249769709568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4249769709568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61274931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237134354854912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42113622 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15498112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4373548120064Ãç øÝ'¹×O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237134249793536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61274931 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4249769709568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4249769709568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61274931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237134354854912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42113622 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15498112 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4373548120064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226823747677184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45958313 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚237134354854912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229864699744176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61285074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844993406718915e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107243389 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4373371900928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107233244 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232884568105984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42113356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15498375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958313 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222450174653440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222450199557120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚2414.583445554 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232884585145344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4249681687552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237134249793536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚6049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958315 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4236397878638 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61285074 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229864682704816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482èð‘Ÿ‘2º- _wandbruntime‚2414  _runtime‚2414 ó•»ÜM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26619534 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4250359797248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4250447826432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61284829 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234138628602658 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45965634 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26619537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107260622 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222485869810688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229901571811936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚6049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232921934473216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107250463 \HDeviceStatsMonitor.on_train_bÄý>ÚÏatch_end/inactive_split_bytes.all.allocated‚226860148997632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222485894714368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4374254283264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61294986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4250447826432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107260622 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234138731790604 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229901554772576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4237073830082 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7844993409288626e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42120132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15500845 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4374067585536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237172399339008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚2414.839900552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2ZkŸCç&¼áM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45965636 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45965636 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42120132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15500845 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4374067585536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237172399339008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚2414.839900552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26619534 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4250359797248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4250447826432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61284829 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234138628602658 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45965634 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26619537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107260622 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222485869810688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229901571811936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚6049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232921934473216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107250463 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226860148997632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222485894714368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4374254283264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61294986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4250447826432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107260622 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234138731790604 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229901554772576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4237073830082 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7844993409288626e+09 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _runtime‚2433.349294483 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234752886467584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4270227354046 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4283619429376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228643255863808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234752903506944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228643467472384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4270313502520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844993594382813e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108096046 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42451690 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4408510052864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4283707451392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224234957419520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235978616634696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231708303132176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234752886467584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61781826 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26829747 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46324363 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46324363 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239036505896960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239036505896960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4283707451392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42451439 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26829744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61771683 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4408323347968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0,ŸëèíÉ×O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26829744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61771683 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4408323347968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _runtime‚2433.349294483 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234752886467584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4270227354046 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4283619429376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228643255863808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234752903506944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228643467472384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4270313502520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844993594382813e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108096046 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42451690 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4408510052864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4283707451392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224234957419520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235978616634696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231708303132176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234752886467584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61781826 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26829747 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46324363 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46324363 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239036505896960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239036505896960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4283707451392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42451439ìÎØE2Ê- _wandbruntime‚2433  _runtime‚2433JŒçÞâ&ËÜM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4284297539072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61781581 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42458465 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42458214 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224270652576768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236016164654012 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231745175199936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4284385568256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4409210148864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4284297539072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4270989454076 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234790269874176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108123424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234790252834816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108123424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46331686 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46331684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228679862725632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4284385568256 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61791738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4409025548288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108113265 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26834037 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46331686 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234790269874176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚2433.608081318 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108113265 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236016061466066 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104¹‹ç&ÌáM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4409025548288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108113265 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26834037 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46331686 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234790269874176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚2433.608081318 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108113265 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236016061466066 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4284297539072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61781581 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42458465 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42458214 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224270652576768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236016164654012 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231745175199936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4284385568256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4409210148864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4284297539072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4270989454076 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234790269874176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108123424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234790252834816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108123424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46331686 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46331684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228679862725632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4284385568256 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61791738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240938762000384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4317557171200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46690415 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237856049498104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236621204829184 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42789544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236621221868544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _step‚496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46690413 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226019715281920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 trainer/global_step‚6149 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240938762000384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4317645193216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237855946310270 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108958848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46690413 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108968993 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27044247 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚233551906520176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240938867061760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4304142977928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62278578 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230463155316224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240938867061760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4317645193216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226019690378240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4443440034304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482)ÅØÜ'×ÖO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108958848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46690413 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108968993 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27044247 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚233551906520176 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240938867061760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4304142977928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62278578 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230463155316224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240938867061760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4317645193216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226019690378240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4443440034304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240938762000384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4317557171200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46690415 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237856049498104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236621204829184 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42789544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236621221868544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _step‚496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46690413 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226019715281920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 trainer/global_step‚6149 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240938762000384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4317645193216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237855946310270 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ðDõéª2Ø- _wandbruntime‚2452  _runtime‚2452ÎÉ[éâ&ÙÜM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4304818929484 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46697734 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27048534 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62278333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚2452.558654097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233588778587936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46697736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚6149 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚497 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62278333 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240976911545856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236658571196416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844993786475708e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236658571196416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230499339366400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226055385535488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108976067 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62288490 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15747735 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226055410439168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237893597517420 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233588761548576 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230499544676352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4443953830912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237893494329474 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4304732780898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240976806477312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108986226 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42796269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108986226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4318235280896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62288490 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108976067 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62288490 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15747735 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226055410439168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237893597517420 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233588761548576 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230499544676352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4443953830912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237893494329474 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4304732780898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240976806477312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108986226 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.smal)vÌj:l_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42796269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108986226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4318235280896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62288490 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62278333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚2452.558654097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233588778587936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46697736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚6149 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚497 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62278333 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240976911545856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236658571196416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844993786475708e+09ñ0‚pà2Û- _wandbruntime‚2452  _runtime‚2452üë4Ü‚ 0loss/FRt1/assignment_loss‚2.3450376276338956 2loss/SRqqt1/assignment_loss‚0.6344640044707127 2loss/SRqqt2/assignment_loss‚0.6670580047505267 /loss/FBt1/detection_loss‚0.5253933941488728 % _timestamp‚1.7844993786485317e+09 _runtime‚2452.559147488 1loss/FBt1/assignment_loss‚0.07043162883202733 1loss/SRqqt2/detection_loss‚0.6875264586987676 trainer/global_step‚6149 _step‚498 0loss/FRt2/assignment_loss‚1.0478446957974286 2loss/FBt2/assignment_loss‚0.004206374501226102 /loss/FRt2/detection_loss‚1.4732301026086443 1loss/SRqqt1/detection_loss‚1.1682230175105912 /loss/FBt2/detection_loss‚0.6836035706335258 %loss/total_loss‚9.910186246725516 epoch‚7 /loss/FRt1/detection_loss‚0.6031673671392983ë§tf ݇ 2loss/FBt2/assignment_loss‚0.004206374501226102 /loss/FRt2/detection_loss‚1.4732301026086443 1loss/SRqqt1/detection_loss‚1.1682230175105912 /loss/FBt2/detection_loss‚0.6836035706335258 %loss/total_loss‚9.910186246725516 epoch‚7 /loss/FRt1/detection_loss‚0.6031673671392983 0loss/FRt1/assignment_loss‚2.3450376276338956 2loss/SRqqt1/assignment_loss‚0.6344640044707127 2loss/SRqqt2/assignment_loss‚0.6670580047505267 /loss/FBt1/detection_loss‚0.5253933941488728 % _timestamp‚1.7844993786485317e+09 _runtime‚2452.559147488 1loss/FBt1/assignment_loss‚0.07043162883202733 1loss/SRqqt2/detection_loss‚0.6875264586987676 trainer/global_step‚6149 _step‚498 0loss/FRt2/assignment_loss‚1.0478446957974286òâöØŠSÞ‚(:È º™õÒÞ€,' proc.memory.availableMB‚ 126.3828125" proc.memory.rssMB‚ 2833.2578125 disk.nvme0c0n1.in‚ 0.0390625 network.sent‚ 8.151088e+06 disk.md0.in‚0 disk.md0.out‚ 67.76171875& disk.nvme0n1.out‚11265.65185546875 cpu‚26.32149565480539 disk.dm-0.in‚ 0.0390625# disk.dm-0.out‚17530.47314453125 disk.dm-1.out‚1474.826171875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚11265.65185546875 network.recv‚740945( disk./.usagePercent‚75.9908244113273% disk./.usageGB‚1166.6491317749023 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme1n1.out‚7884.8408203125) proc.memory.percent‚17.29283332824707 disk.nvme0n1.in‚ 0.0390625& disk.nvme1c1n1.out‚7884.8408203125# memory_percent‚99.2286205291748`…Jnß‚(:„ º™õÒÞµ­, gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.447' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.149 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…í™2à- _wandbruntime‚2471  _runtime‚2471.2dÆáÀ & lr-AdamW/pg1‚9.708171807305322e-05 & lr-AdamW/pg2‚9.708171807305322e-05 trainer/global_step‚6199 $ _timestamp‚1.784499397295546e+09 _runtime‚2471.207596671 _step‚499£ê.^ËâÅ trainer/global_step‚6199 $ _timestamp‚1.784499397295546e+09 _runtime‚2471.207596671 _step‚499 & lr-AdamW/pg1‚9.708171807305322e-05 & lr-AdamW/pg2‚9.708171807305322e-05ógòtÓ2ã- _wandbruntime‚2471  _runtime‚2471p,¶<Ø'äÒO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238489523190784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242841018103808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚500 trainer/global_step‚6199 % _timestamp‚1.7844993972961524e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232282865069568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4351494913024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62775330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4351582935040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43127953 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235395492868816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109831795 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4337972453336 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235395509908176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238489523190784 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109821650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239733379173678 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47056465 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47056463 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47056463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15868950 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242841018103808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232282651363840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227804448240640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4351582935040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62775330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238489540230144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235395509908176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238489523190784 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109821650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239733379173678 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47056465 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47056463 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47056463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15868950 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242841018103808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232282651363840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227804448240640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4351582935040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62775330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238489540230144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4351494913024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62775330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4351582935040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152ôûkß 2æ- _wandbruntime‚2471  _runtime‚2471ÉÛ®0_ çÜM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232319048307712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47063784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242879167649280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚6199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109849028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239770927192882 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47063786 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62785242 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47063784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109849028 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_5ÚV8ƒpool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238526906597376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352173022720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352261051904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47063786 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227840168301568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚501 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27263034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4479081121792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238526906597376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227840143397888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62775085 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352173022720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239771030380828 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4338562256306 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109838869 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238526889558016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235432381975936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242879062580736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238526889558016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62785242 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43134410 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4338648404892 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43134668 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27263037 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4478904909824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15871631 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232319249423360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844993975552218e+09 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47063784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242879167649280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚6199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109849028 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239770927192882 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47063786 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62785242 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47063784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109849028 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238526906597376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352173022720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352261051904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47063786 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227840168301568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚501 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27263034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4479081121792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238526906597376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227840143397888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62775085 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352173022720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239771030380828 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4338562256306 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109838869 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238526889558016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235432381975936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242879062580736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238526889558016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62785242 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43134410 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4338648404892 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43134668 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27263037 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4478904909824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15871631 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232319249423360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816õÇR·2é- _wandbruntime‚2471  _runtime‚2471ˆ¾Ì êƒ .loss/FRt1/assignment_loss‚2.50977869691089 0loss/FRt2/assignment_loss‚0.9884340368971533 3loss/FBt1/assignment_loss‚0.0060543557211443855 2loss/FBt2/assignment_loss‚0.010204677486773802 /loss/FRt1/detection_loss‚0.4796949047975382 /loss/FRt2/detection_loss‚1.4681634364269522 /loss/FBt1/detection_loss‚2.4064381908226227 &loss/total_loss‚11.208698520783855 2loss/SRqqt2/assignment_loss‚0.6417401400931523 1loss/SRqqt1/detection_loss‚0.8945626873961267 epoch‚7 trainer/global_step‚6199 _runtime‚2471.466892147 1loss/SRqqt2/detection_loss‚0.6455041179742117 /loss/FBt2/detection_loss‚0.4020036441986175 % _timestamp‚1.7844993975561786e+09 _step‚502 2loss/SRqqt1/assignment_loss‚0.7561196320586734!uø·ëˆ .loss/FRt1/assignment_loss‚2.50977869691089 0loss/FRt2/assignment_loss‚0.9884340368971533 3loss/FBt1/assignment_loss‚0.0060543557211443855 2loss/FBt2/assignment_loss‚0.010204677486773802 /loss/FRt1/detection_loss‚0.4796949047975382 /loss/FRt2/detection_loss‚1.4681634364269522 /loss/FBt1/detection_loss‚2.4064381908226227 &loss/total_loss‚11.208698520783855 2loss/SRqqt2/assignment_loss‚0.6417401400931523 1loss/SRqqt1/detection_loss‚0.8945626873961267 epoch‚7 trainer/global_step‚6199 _runtime‚2471.466892147 1loss/SRqqt2/detection_loss‚0.6455041179742117 /loss/FBt2/detection_loss‚0.4020036441986175 % _timestamp‚1.7844993975561786e+09 _step‚502 2loss/SRqqt1/assignment_loss‚0.7561196320586734öÉïí¢Sì‚(:È É™õÒëìê+$ disk.nvme1n1.out‚7885.3720703125 disk.dm-0.in‚ 0.0390625 disk.dm-1.in‚0 disk.dm-1.out‚1474.884765625& disk.nvme1c1n1.out‚7885.3720703125 cpu‚26.319159221219522 network.recv‚745634) disk./.usagePercent‚75.99082366590525 disk.nvme1n1.in‚0# disk.dm-0.out‚17531.17236328125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚11266.58935546875& disk.nvme0n1.out‚11266.58935546875 disk.nvme1c1n1.in‚0 network.sent‚ 8.213616e+06 disk.nvme0c0n1.in‚ 0.0390625" memory_percent‚99.228835105896( proc.memory.availableMB‚ 126.34765625" proc.memory.rssMB‚ 2833.2578125 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 68.078125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.0390625) proc.memory.percent‚17.29283332824707% disk./.usageGB‚1166.6491203308105Ïxší‚(: É™õÒ±˜, gpu.0.gpu‚75.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚138.962' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚46.32066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ŸòJî‚(:¿ Ø™õÒÝÀÜ+% proc.memory.availableMB‚ 126.34375" proc.memory.rssMB‚ 2833.2578125 network.sent‚ 8.263067e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 1475.01171875& disk.nvme1c1n1.out‚7885.8974609375 disk.md0.in‚0 cpu‚26.316849643012837$ disk./.usageGB‚1166.649299621582 disk.nvme0c0n1.in‚ 0.0390625& disk.nvme0c0n1.out‚11268.330078125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚7885.8974609375# memory_percent‚99.2288589477539 disk.dm-0.in‚ 0.0390625 disk.nvme0n1.in‚ 0.0390625$ disk.nvme0n1.out‚11268.330078125 disk.md0.out‚ 68.296875) proc.memory.percent‚17.29283332824707 proc.cpu.threads‚17 network.recv‚749834) disk./.usagePercent‚75.99083534418382" disk.dm-0.out‚17532.8271484375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0k/Ø‚ï‚(:„ Ø™õÒñŠ”, gpu.0.gpu‚57.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.963' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.321 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€s~^2ð- _wandbruntime‚2490  _runtime‚2490bÕ¡äÇñÁ & lr-AdamW/pg1‚9.699608073227836e-05 & lr-AdamW/pg2‚9.699608073227836e-05 trainer/global_step‚6249 % _timestamp‚1.7844994162249944e+09 _runtime‚2490.137377386 _step‚503WHÌòÆ _step‚503 & lr-AdamW/pg1‚9.699608073227836e-05 & lr-AdamW/pg2‚9.699608073227836e-05 trainer/global_step‚6249 % _timestamp‚1.7844994162249944e+09 _runtime‚2490.137377386÷ôç2ó- _wandbruntime‚2490  _runtime‚2490¿0ùUØ'ôÒO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7844994162256453e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240357841552384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244743274207232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241610812037086 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237239096256816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47422515 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385520676864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43465082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43464827 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27473247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241610915224920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385432654848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229589206103040 _runtime‚2490.137440046 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15991583 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229589231006720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240357858591744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234102367335424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47422513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15991835 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240357858591744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63261939 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27473244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244743274207232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385520676864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234102578944000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4513347937280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63272082 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63272082 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4513161232384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47422515 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4371715780270 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240357841552384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ÞgOõ×O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4385432654848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMoni &§Ž&tor.on_train_batch_start/requested_bytes.all.allocated‚241610915224920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385432654848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229589206103040 _runtime‚2490.137440046 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15991583 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229589231006720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240357858591744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234102367335424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47422513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15991835 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240357858591744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63261939 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27473244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244743274207232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385520676864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234102578944000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4513347937280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63272082 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63272082 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4513161232384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47422515 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4371715780270 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240357841552384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7844994162256453e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240357841552384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244743274207232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241610812037086 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237239096256816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47422515 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385520676864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43465082 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43464827 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27473247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63271837 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43471547 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4386198793728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110701671 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844994164854858e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386110764544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237275968324576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110711830 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47429834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43471789 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234138771368448 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27477534 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27477537 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240395207919616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237275985363936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4372391731714 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244781423752704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241648463244236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15994013 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229624901260288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚6249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240395207919616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15994252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229624926163968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244781423752704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4386198793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47429834 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63271837 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47429836 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4513870108160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63281994 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544/4&²ç&øáM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63271837 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43471547 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4386198793728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110701671 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844994164854858e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386110764544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237275968324576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110711830 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47429834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43471789 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234138771368448 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27477534 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27477537 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240395207919616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237275985363936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4372391731714 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244781423752704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241648463244236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15994013 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229624901260288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚6249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240395207919616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15994252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229624926163968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244781423752704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4386198793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47429834 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63271837 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47429836 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4513870108160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63281994 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816ù1Áp2ù- _wandbruntime‚2490  _runtime‚2490Æ÷úÿ /loss/FRt1/assignment_loss‚2.376811522080893 2loss/SRqqt2/assignment_loss‚0.6478297572236691 1loss/SRqqt1/detection_loss‚1.1088449059814018 /loss/FBt1/detection_loss‚1.8856112152071935 epoch‚7 $ _timestamp‚1.784499416486425e+09 _step‚506 /loss/FRt2/assignment_loss‚1.075991412003835 2loss/FBt2/assignment_loss‚0.006287871307185631 /loss/FRt1/detection_loss‚0.5952504586094791 /loss/FRt2/detection_loss‚1.3393904972536903 0loss/SRqqt2/detection_loss‚0.675032990148545 &loss/total_loss‚11.144872895381184 trainer/global_step‚6249 1loss/FBt1/assignment_loss‚0.05632223680284402 /loss/FBt2/detection_loss‚0.6595175986012984 _runtime‚2490.397038832 2loss/SRqqt1/assignment_loss‚0.7179824301611485ÆÊV© û„ 2loss/FBt2/assignment_loss‚0.006287871307185631 /loss/FRt1/detection_loss‚0.5952504586094791 /loss/FRt2/detection_loss‚1.3393904972536903 0loss/SRqqt2/detection_loss‚0.675032990148545 &loss/total_loss‚11.144872895381184 trainer/global_step‚6249 1loss/FBt1/assignment_loss‚0.05632223680284402 /loss/FBt2/detection_loss‚0.6595175986012984 _runtime‚2490.397038832 2loss/SRqqt1/assignment_loss‚0.7179824301611485 /loss/FRt1/assignment_loss‚2.376811522080893 2loss/SRqqt2/assignment_loss‚0.6478297572236691 1loss/SRqqt1/detection_loss‚1.1088449059814018 /loss/FBt1/detection_loss‚1.8856112152071935 epoch‚7 $ _timestamp‚1.784499416486425e+09 _step‚506 /loss/FRt2/assignment_loss‚1.075991412003835ú/ì½Rü‚(:Ç ç™õÒüØ—, disk.nvme1n1.in‚0 disk.nvme0n1.in‚0.09375 disk.nvme1c1n1.in‚0) proc.memory.percent‚17.29283332824707 network.sent‚ 8.281765e+06 network.recv‚753176$ disk./.usageGB‚1168.156681060791 disk.md0.in‚0( disk.nvme0c0n1.out‚11335.45068359375' disk.nvme1c1n1.out‚7894.61962890625# memory_percent‚99.2290735244751" proc.memory.rssMB‚ 2833.2578125% disk.nvme1n1.out‚7894.61962890625 disk.md0.out‚ 71.0234375" disk.dm-1.out‚1475.55712890625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.09375 proc.cpu.threads‚17) disk./.usagePercent‚76.08902009840685 disk.dm-0.in‚0.09375# disk.dm-0.out‚17602.59912109375 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚11335.45068359375( proc.memory.availableMB‚ 126.30859375 cpu‚26.314753494965508˜[wî™ý‚(:Ž ç™õÒîËÚ+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.689' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.56299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mû“@þ-tI- _wandbruntime‚2508  _runtime‚2508#åw3ÆÿÀ & lr-AdamW/pg1‚9.690970472826894e-05 & lr-AdamW/pg2‚9.690970472826894e-05 trainer/global_step‚6299 $ _timestamp‚1.784499434815835e+09 _runtime‚2508.728411891 _step‚507a±êˀŠ& lr-AdamW/pg2‚9.690970472826894e-05 trainer/global_step‚6299 $ _timestamp‚1.784499434815835e+09 _runtime‚2508.728411891 _step‚507 & lr-AdamW/pg1‚9.690970472826894e-05ûhÈJ2- _wandbruntime‚2508  _runtime‚2508[U³ÏØ'‚ÒO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246645530310656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242226159913984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4548139106816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16114570 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111557399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231373988869120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243488348088328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27687744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43802314 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63768834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27687747 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235922308389376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231373963965440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 % _timestamp‚1.7844994348164735e+09 _runtime‚2508.728475471 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16114808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239082716684176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242226159913984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246645530310656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242226176953344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243488244900494 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246645635372032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4419370396672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4405631404152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4405545255678 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63758691 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4419458418688 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4419370396672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63768834 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239082699644816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143e4¸…Ý'ƒ×O `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242226176953344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243488244900494 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246645635372032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4419370396672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4405631404152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4405545255678 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63758691 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4419458418688 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4419370396672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63768834 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239082699644816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246645530310656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242226159913984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4548139106816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16114570 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111557399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231373988869120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243488348088328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27687744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43802314 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63768834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27687747 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235922308389376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231373963965440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 % _timestamp‚1.7844994348164735e+09 _runtime‚2508.728475471 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16114808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239082716684176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242226159913984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246645530310656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚508üôQá 2„- _wandbruntime‚2508  _runtime‚2508¢/ì7â&…ÜM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242263526281216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243525896107644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242263526281216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47795884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235958489740288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111574632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27692034 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242263543320576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43808981 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111564473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4420136535552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111564473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4406221207122 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚6299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4420048506368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231409684026368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231409659122688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4548830617600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243525792919698 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63778746 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63768589 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111574632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47795886 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4406307355708 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚08r¨†áM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4549017315328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63768589 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43808981 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111564473 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4420136535552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111564473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4406221207122 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚6299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4420048506368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pooñ$ûßl.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231409684026368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231409659122688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4548830617600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243525792919698 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63778746 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63768589 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111574632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47795886 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4406307355708 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242263526281216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243525896107644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242263526281216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47795884 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235958489740288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111574632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27692034 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242263543320576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248547786414080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48154615 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48154615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4453308138496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245365677763902 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244094495314944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48154613 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844994537121422e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4453308138496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16237860 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _step‚512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248547891475456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27902244 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4453396160512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64265586 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44140104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244094478275584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27902247 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4439374731086 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244094495314944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237742023206400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4583096061440 _runtime‚2527.626362238 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244094478275584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4453396160512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112420201 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚233158746731520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248547786414080 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16238111 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4583276474880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64255443 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4439374731086 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244094495314944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237742023206400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4583096061440 _runtime‚2527.626362238 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244094478275584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4453396160512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112420201 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚233158746731520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248547786414080 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16238111 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4583276474880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64255443 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248547786414080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48154615 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48154615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4453308138496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245365677763902 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244094495314944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48154613 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844994537121422e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4453308138496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16237860 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _step‚512 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248547891475456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27902244 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4453396160512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64265586 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44140104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244094478275584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27902247 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233194416985088 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7844994539695826e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4440136831116 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16240223 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4583971898368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240963192139936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44146757 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27906534 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248585935959552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16240467 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64275498 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64265341 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27906537 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4440050682530 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48161936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4583787297792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240963175100576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44147004 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112437434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48161934 trainer/global_step‚6349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚248585935959552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚2527.880632538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4454074277376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4453986248192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244131844642816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112437434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248585830891008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚513@88[ç&”áM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4454074277376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4453986248192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244131844642816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237778413787136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245403328971052 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4454074277376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4453986248192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244131844642816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112437434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248585830891008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚513 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233194416985088 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7844994539695826e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4440136831116 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16240223 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4583971898368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240963192139936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44146757 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27906534 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248585935959552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16240467 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64275498 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64265341 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27906537 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4440050682530 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48161936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4583787297792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240963175100576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44147004 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112437434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48161934 trainer/global_step‚6349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚248585935959552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚2527.880632538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245962813676544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247243213815144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4473290354968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487245880320 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487333902336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234943504593920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4473204206494 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2546.559404361 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239561517312000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242769906420816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7844994726447375e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487333902336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113283003 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234943479690240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64762338 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250450147578880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16361850 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64752195 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113272858 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44478597 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚6399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48520665 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247243110627310 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64762338 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44478335 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16361591 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250450147578880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245962796637184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4618226423808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250450042517504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487245880320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4618037621760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245962813676544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250450042517504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28116744 =6DeviceStatsMonçT¸³oitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482§PD^Ý'Ÿ×O VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250450147578880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245962796637184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4618226423808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250450042517504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487245880320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4618037621760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245962813676544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250450042517504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28116744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239561731017728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64752195 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242769923460176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245962813676544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247243213815144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4473290354968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487245880320 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487333902336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234943504593920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4473204206494 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2546.559404361 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239561517312000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242769906420816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7844994726447375e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487333902336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113283003 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234943479690240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64762338 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250450147578880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16361850 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64752195 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113272858 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44478597 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚6399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48520665 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247243110627310 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64762338 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44478335 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16361591 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145„þÉ’2 - _wandbruntime‚2546  _runtime‚2546<Ç áâ&¡ÜM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113290077 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488012019200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242806778488576 _step‚517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48527984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239598123077120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64772250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113300236 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16364033 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2546.813159502 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113300236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64762093 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4487923990016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4473966306524 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250488086994432 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246000180043776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246000163004416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234979174847488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44485309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239597907281408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4618923325952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7844994729021099e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234979199751168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242806795527936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246000180043776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247280761834460 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48527986 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚6399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246000163004416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250488192062976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44485067 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16364272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4473966306524 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250488086994432 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246000180043776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246000163004416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234979174847488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44485309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239597907281408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4618923325952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7844994729021099e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234979199751168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242806795527936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246000180043776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247280761834460 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48527986 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚6399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246000163004416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250488192062976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44485067 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16364272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488012019200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242806778488576 _step‚517 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48527984 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239598123077120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64772250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113300236 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16364033 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2546.813159502…k‰PØ2£- _wandbruntime‚2546  _runtime‚2546€”䤀 2loss/SRqqt2/assignment_loss‚0.6470349218874363 1loss/SRqqt1/detection_loss‚1.1877139364154292 1loss/SRqqt2/detection_loss‚0.7624657594945369 /loss/FBt1/detection_loss‚1.1562846349552274 _runtime‚2546.813813623 _step‚518 /loss/FRt1/assignment_loss‚2.505602484467638 /loss/FBt2/detection_loss‚0.8708335642121752 &loss/total_loss‚11.220247139260625 epoch‚7 $ _timestamp‚1.784499472903037e+09 2loss/SRqqt1/assignment_loss‚0.7197086368060599 /loss/FRt2/detection_loss‚1.5087363802213005 0loss/FRt2/assignment_loss‚1.2139685761514318 1loss/FBt1/assignment_loss‚0.05558356713785228 1loss/FBt2/assignment_loss‚0.03154806416244934 /loss/FRt1/detection_loss‚0.5607666133490886 trainer/global_step‚6399TÕÁa ¥… /loss/FRt1/detection_loss‚0.5607666133490886 trainer/global_step‚6399 2loss/SRqqt2/assignment_loss‚0.6470349218874363 1loss/SRqqt1/detection_loss‚1.1877139364154292 1loss/SRqqt2/detection_loss‚0.7624657594945369 /loss/FBt1/detection_loss‚1.1562846349552274 _runtime‚2546.813813623 _step‚518 /loss/FRt1/assignment_loss‚2.505602484467638 /loss/FBt2/detection_loss‚0.8708335642121752 &loss/total_loss‚11.220247139260625 epoch‚7 $ _timestamp‚1.784499472903037e+09 2loss/SRqqt1/assignment_loss‚0.7197086368060599 /loss/FRt2/detection_loss‚1.5087363802213005 0loss/FRt2/assignment_loss‚1.2139685761514318 1loss/FBt1/assignment_loss‚0.05558356713785228 1loss/FBt2/assignment_loss‚0.03154806416244934† #›¦‚(: ”šõÒ½ƒ´, gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚135.727' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚45.242333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿þÿP•§‚(:½ ”šõÒ†žî- disk.nvme1n1.in‚1.625 disk.md0.in‚0 disk.md0.out‚74.125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚12663.4482421875 network.recv‚765751% disk.nvme0n1.out‚12663.4482421875 disk.dm-0.in‚9# disk.dm-0.out‚20295.25244140625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚7.375( proc.memory.availableMB‚ 126.17578125" proc.memory.rssMB‚ 2833.2578125k€ ³ network.sent‚ 8.455068e+06 disk.nvme0n1.in‚7.375% disk./.usageGB‚1169.3146781921387$ disk.nvme1n1.out‚9266.6240234375" disk.dm-1.out‚1476.50634765625 disk.rbd0.in‚0$ memory_percent‚99.22988414764404) proc.memory.percent‚17.29283332824707 cpu‚26.30821602234164 proc.cpu.threads‚17( disk./.usagePercent‚76.1644473663663 disk.nvme1c1n1.in‚1.625& disk.nvme1c1n1.out‚9266.6240234375\R…S¨‚(:È £šõÒ‹­Ñ+ disk.rbd0.out‚0( disk.nvme0c0n1.out‚12664.33642578125 disk.nvme1c1n1.in‚1.625' disk.nvme1c1n1.out‚9267.24658203125 disk.dm-0.in‚9( proc.memory.availableMB‚ 107.33984375 disk.dm-1.in‚0" disk.dm-1.out‚1476.62744140625) proc.memory.percent‚17.40567684173584 cpu‚26.324894774091657 disk.nvme1n1.in‚1.625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚7.375 disk.nvme0n1.in‚7.375& disk.nvme0n1.out‚12664.33642578125 disk.md0.out‚ 74.3828125$ memory_percent‚99.34484958648682 network.sent‚ 8.516226e+06 network.recv‚770275) disk./.usagePercent‚76.16438897497353% disk./.usageGB‚1169.3137817382812 disk.md0.in‚0# disk.dm-0.out‚20296.06103515625# proc.memory.rssMB‚ 2851.74609375 proc.cpu.threads‚17% disk.nvme1n1.out‚9267.24658203125pÙÀ˜©‚(: £šõÒÈÀ, gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.3' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.43333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é´½2ª- _wandbruntime‚2565  _runtime‚2565Gœ¤ê¤=«ž{ ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16802939772605896 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14685463905334473 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 _runtime‚ 2565.02607638 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08011745661497116 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6078852415084839 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12311793863773346 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21792927384376526 .particle/accuracy_3_of_3‚0.999514102935791 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03462669253349304 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9918051362037659 (particle/accuracy‚0.8496636152267456 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 jet/accuracy_3_of_5‚1 trainer/global_step‚6407 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6561939120292664 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13715903460979462 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.841823399066925 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9945542216300964 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44776248931884766 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14104700088500977 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5434542894363403 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03654079884290695 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_1‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7072649002075195 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1587664783000946 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 .particle/accuracy_3_of_4‚0.995192289352417 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9942029118537903 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7337291836738586 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799453020095825 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9619818329811096 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 +particle/specificity‚0.9111273884773254 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6120305061340332 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7573880553245544 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9630549550056458 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12128797918558121 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5451388955116272 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6916511654853821 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006720640696585178 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6625746488571167 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030409129336476326 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10781250149011612 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13946540653705597 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7580409646034241 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02706443890929222 +jet/accuracy_2_of_2‚0.13581474125385284 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8118159770965576 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.796062171459198 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5765177607536316 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1872633397579193 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987359046936035 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5294707417488098 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5975424647331238 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922944903373718 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6068224310874939 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014021163806319237 *jet/accuracy_1_of_3‚0.9104477167129517 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7344655990600586 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3366334140300751 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5019552707672119 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1600751429796219 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7279847860336304 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.382883757352829 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01071428507566452 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01327941007912159 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7728069424629211 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9033597111701965 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37262022495269775 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46411022543907166 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1712285876274109 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5387105941772461 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8030537962913513 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611111044883728 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992781400680542 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6034213304519653 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9793288111686707 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8435625433921814 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6330252289772034 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06543532013893127 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7279661893844604 epoch‚7 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9809908866882324 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6068224310874939 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12289241701364517 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44273585081100464 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,jet/accuracy_3_of_3‚0.061747487634420395 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35682085156440735 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.034971706569194794 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978895783424377 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11338428407907486 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09834016859531403 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6859978437423706 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47820213437080383 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13694950938224792 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.728507936000824 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025813845917582512 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1603846549987793 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7351318001747131 *jet/accuracy_2_of_3‚0.5292706489562988 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5987923741340637 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3722408711910248 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7185401916503906 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936221241950989 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.29021573066711426 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 particle/accuracy_2_of_5‚1 % _timestamp‚1.7844994911148582e+09 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9974656701087952 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9939153790473938 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3901515305042267 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972757697105408 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13901051878929138 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6692349314689636 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14685463905334473 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )loss/val_total_loss‚9.700491178597881 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 /particle/accuracy_2_of_2‚0.9998950958251953 jet/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8927980065345764 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6629519462585449 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9055883288383484 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14472541213035583 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03573904559016228 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9903846383094788 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9901315569877625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5845160484313965 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8555783629417419 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8555783629417419 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9979138374328613 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03872611001133919 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7868419885635376 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2638888657093048 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8008546233177185 +particle/sensitivity‚0.6248828768730164 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49075543880462646 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1680247038602829 /particle/accuracy_4_of_4‚0.9529106616973877 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2713373899459839 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5837185978889465 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6409024000167847 particle/accuracy_3_of_5‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02706443890929222 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5951722264289856 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4921875298023224 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11968187987804413 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5313742160797119 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.601320743560791 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3567163944244385 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12463787943124771 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08302710205316544 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21790730953216553 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 *jet/accuracy_1_of_2‚0.7482993602752686 +jet/accuracy_3_of_4‚0.09205285459756851 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7874058485031128 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚519 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23263390362262726 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8016840815544128 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9823423027992249 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6002715229988098 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6063917875289917 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0~ÆKJb;¬£{ 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14104700088500977 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5434542894363403 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03654079884290695 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_1‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7072649002075195 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1587664783000946 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 .particle/accuracy_3_of_4‚0.995192289352417 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9942029118537903 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7337291836738586 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9799453020095825 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9619818329811096 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 +particle/specificity‚0.9111273884773254 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6120305061340332 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7573880553245544 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9630549550056458 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12128797918558121 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5451388955116272 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6916511654853821 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006720640696585178 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6625746488571167 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030409129336476326 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10781250149011612 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13946540653705597 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7580409646034241 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02706443890929222 +jet/accuracy_2_of_2‚0.13581474125385284 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8118159770965576 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.796062171459198 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5765177607536316 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1872633397579193 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987359046936035 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5294707417488098 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5975424647331238 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922944903373718 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6068224310874939 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014021163806319237 *jet/accuracy_1_of_3‚0.9104477167129517 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7344655990600586 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3366334140300751 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5019552707672119 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1600751429796219 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7279847860336304 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.382883757352829 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01071428507566452 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01327941007912159 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7728069424629211 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9033597111701965 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37262022495269775 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46411022543907166 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1712285876274109 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5387105941772461 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8030537962913513 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611111044883728 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992781400680542 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6034213304519653 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9793288111686707 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8435625433921814 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6330252289772034 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06543532013893127 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7279661893844604 epoch‚7 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9809908866882324 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6068224310874939 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12289241701364517 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44273585081100464 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,jet/accuracy_3_of_3‚0.061747487634420395 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35682085156440735 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.034971706569194794 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978895783424377 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11338428407907486 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09834016859531403 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6859978437423706 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47820213437080383 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13694950938224792 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.728507936000824 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025813845917582512 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1603846549987793 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7351318001747131 *jet/accuracy_2_of_3‚0.5292706489562988 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5987923741340637 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3722408711910248 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7185401916503906 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936221241950989 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.29021573066711426 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 particle/accuracy_2_of_5‚1 % _timestamp‚1.7844994911148582e+09 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9974656701087952 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9939153790473938 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3901515305042267 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972757697105408 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13901051878929138 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6692349314689636 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14685463905334473 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )loss/val_total_loss‚9.700491178597881 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 /particle/accuracy_2_of_2‚0.9998950958251953 jet/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8927980065345764 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6629519462585449 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9055883288383484 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14472541213035583 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03573904559016228 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9903846383094788 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9901315569877625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5845160484313965 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8555783629417419 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8555783629417419 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9979138374328613 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03872611001133919 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7868419885635376 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2638888657093048 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8008546233177185 +particle/sensitivity‚0.6248828768730164 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49075543880462646 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1680247038602829 /particle/accuracy_4_of_4‚0.9529106616973877 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2713373899459839 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5837185978889465 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6409024000167847 particle/accuracy_3_of_5‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02706443890929222 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5951722264289856 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4921875298023224 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11968187987804413 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5313742160797119 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.601320743560791 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3567163944244385 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12463787943124771 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08302710205316544 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21790730953216553 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 *jet/accuracy_1_of_2‚0.7482993602752686 +jet/accuracy_3_of_4‚0.09205285459756851 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7874058485031128 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚519 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23263390362262726 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8016840815544128 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9823423027992249 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6002715229988098 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6063917875289917 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16802939772605896 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14685463905334473 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 _runtime‚ 2565.02607638 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08011745661497116 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6078852415084839 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12311793863773346 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21792927384376526 .particle/accuracy_3_of_3‚0.999514102935791 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03462669253349304 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9918051362037659 (particle/accuracy‚0.8496636152267456 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 jet/accuracy_3_of_5‚1 trainer/global_step‚6407 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.656`øhFG1939120292664 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13715903460979462 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.841823399066925 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9945542216300964 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44776248931884766 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1‡Û- Z­‚(:Ï ²šõÒ‹Ûâ+# proc.memory.rssMB‚ 2851.74609375 cpu‚26.32281963634345 network.sent‚ 8.535065e+06" disk.dm-1.out‚1476.77197265625 disk.rbd0.in‚0& disk.nvme0n1.out‚12668.06103515625$ memory_percent‚99.34446811676025 proc.cpu.threads‚17 network.recv‚773451% disk./.usageGB‚1169.3169784545898 disk.md0.in‚0 disk.md0.out‚ 74.6953125 disk.nvme0c0n1.in‚ 7.37890625( disk.nvme0c0n1.out‚12668.06103515625 disk.nvme1c1n1.in‚1.625) disk./.usagePercent‚76.16459719619539" disk.dm-0.out‚20301.6689453125 disk.rbd0.out‚0 disk.nvme1n1.out‚ 9269.953125) proc.memory.percent‚17.40567684173584 disk.dm-0.in‚ 9.00390625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 7.37890625" disk.nvme1c1n1.out‚ 9269.953125 disk.nvme1n1.in‚1.625( proc.memory.availableMB‚ 107.40234375–<%›®‚(: ²šõÒ€ÚŠ, gpu.0.gpu‚81.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚194.325' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.77499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û(ºß2¯- _wandbruntime‚2580  _runtime‚25809¬JÖÇ°Á & lr-AdamW/pg2‚9.664615863072643e-05 trainer/global_step‚6449 % _timestamp‚1.7844995067215877e+09 _runtime‚2580.634359317 _step‚520 & lr-AdamW/pg1‚9.664615863072643e-05íEÞ‹Ì±Æ _step‚520 & lr-AdamW/pg1‚9.664615863072643e-05 & lr-AdamW/pg2‚9.664615863072643e-05 trainer/global_step‚6449 % _timestamp‚1.7844995067215877e+09 _runtime‚2580.634359317ˆ$I92²- _wandbruntime‚2580  _runtime‚2580º9XÐØ'³ÒO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4673694312448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114540389 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4541603936768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248803778593792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28425327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49124183 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4527427689463 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844995067221053e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6449 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4527513837937 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237653685720064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114550534 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65426351 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4541603936768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4673511801856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4541691958784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242327380032512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237653660816384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242327172618240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245581999277168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65426351 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚248803795633152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253345382530560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44993618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16568291 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4541691958784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65416208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253345487591936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49124181 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253345487591936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253345382530560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44993618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16568291 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4541691958784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65416208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253345487591936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49124181 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253345487591936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114540389 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4541603936768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248803778593792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28425327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49124183 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4527427689463 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844995067221053e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1‰ßÁ##2µ- _wandbruntime‚2580  _runtime‚2580ØzЙ&¶ÜM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253383532076032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114557608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16570753 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65426106 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚248841144961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114557608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237689355973632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248841144961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114567767 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45000370 _step‚522 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4542370075648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16570495 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250146974985835 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4674391280128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65436263 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248841162000384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49131504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6449 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28429617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65426106 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4542282046464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28429614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49131504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114567767 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245618888384288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4528189789493 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4542370075648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253383532076032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45000109 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242363571041792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237689380877312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49131502 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248841162000384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4528103640907 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚2580.891247135 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242363772157440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WIDeviceStatsMo6®ÖIonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104é ñç&·áM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248841162000384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4528103640907 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚2580.891247135 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242363772157440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253383532076032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114557608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16570753 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65426106 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚248841144961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114557608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237689355973632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248841144961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114567767 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45000370 _step‚522 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4542370075648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16570495 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250146974985835 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4674391280128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65436263 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248841162000384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49131504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6449 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28429617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65426106 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4542282046464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28429614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49131504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114567767 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245618888384288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4528189789493 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4542370075648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253383532076032 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45000109 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242363571041792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237689380877312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49131502 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157Š]5o]2¸- _wandbruntime‚2580  _runtime‚2580ÐöQ±¹‚ 0loss/FRt1/assignment_loss‚2.3162053672272704 _runtime‚2580.891708495 _step‚523 0loss/FRt2/assignment_loss‚0.9992450270531834 2loss/SRqqt2/assignment_loss‚0.5767409577744451 1loss/SRqqt1/detection_loss‚1.1514212615754962 % _timestamp‚1.7844995069811208e+09 1loss/SRqqt1/assignment_loss‚0.664021250348167 2loss/FBt1/assignment_loss‚0.016460438193696803 .loss/FBt1/detection_loss‚1.113803289597854 /loss/FBt2/detection_loss‚0.5525144662834239 epoch‚8 trainer/global_step‚6449 3loss/FBt2/assignment_loss‚0.0032402500665436188 /loss/FRt1/detection_loss‚0.5430434042212883 /loss/FRt2/detection_loss‚1.3476359982758868 1loss/SRqqt2/detection_loss‚0.6460114352907934 %loss/total_loss‚9.930343145908049i³º÷ º‡ 0loss/FRt2/assignment_loss‚0.9992450270531834 2loss/SRqqt2/assignment_loss‚0.5767409577744451 1loss/SRqqt1/detection_loss‚1.1514212615754962 % _timestamp‚1.7844995069811208e+09 1loss/SRqqt1/assignment_loss‚0.664021250348167 2loss/FBt1/assignment_loss‚0.016460438193696803 .loss/FBt1/detection_loss‚1.113803289597854 /loss/FBt2/detection_loss‚0.5525144662834239 epoch‚8 trainer/global_step‚6449 3loss/FBt2/assignment_loss‚0.0032402500665436188 /loss/FRt1/detection_loss‚0.5430434042212883 /loss/FRt2/detection_loss‚1.3476359982758868 1loss/SRqqt2/detection_loss‚0.6460114352907934 %loss/total_loss‚9.930343145908049 0loss/FRt1/assignment_loss‚2.3162053672272704 _runtime‚2580.891708495 _step‚523‹Ü¼Àb»‚(:× ÁšõÒåǦ+ disk.dm-0.in‚ 9.00390625 disk.nvme0n1.in‚ 7.37890625% disk.nvme1n1.out‚9270.64306640625" disk.dm-0.out‚20302.2119140625 disk.dm-1.in‚0" disk.dm-1.out‚1476.91259765625 disk.rbd0.out‚0$ memory_percent‚99.34470653533936 cpu‚26.32058951334812 network.sent‚ 8.578173e+06 disk.nvme1c1n1.in‚1.625 disk.md0.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 107.36328125# proc.memory.rssMB‚ 2851.74609375) proc.memory.percent‚17.40567684173584 disk.nvme0c0n1.in‚ 7.37890625 disk.md0.out‚ 75.12109375 proc.cpu.threads‚17 network.recv‚778013) disk./.usagePercent‚76.16457682132643$ disk./.usageGB‚1169.316665649414' disk.nvme0c0n1.out‚12668.9814453125% disk.nvme0n1.out‚12668.9814453125' disk.nvme1c1n1.out‚9270.64306640625 disk.nvme1n1.in‚1.625]Λ›¼‚(: ÁšõÒþ¸ø+ gpu.0.gpu‚15.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.786' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.595333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NKe…2½- _wandbruntime‚2599  _runtime‚2599ê&ǾÁ _runtime‚2599.521987197 _step‚524 & lr-AdamW/pg1‚9.655684192662743e-05 & lr-AdamW/pg2‚9.655684192662743e-05 trainer/global_step‚6499 % _timestamp‚1.7844995256110396e+09fõûÌ¿Æ _step‚524 & lr-AdamW/pg1‚9.655684192662743e-05 & lr-AdamW/pg2‚9.655684192662743e-05 trainer/global_step‚6499 % _timestamp‚1.7844995256110396e+09 _runtime‚2599.521987197Œ=¦ô†2À- _wandbruntime‚2599  _runtime‚2599Ih(Î×'ÁÑO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251986859830039 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255247638633984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115403191 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45331841 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239438443582464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 $ _timestamp‚1.784499525611995e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49490233 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255247638633984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65912960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4575629700608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4575541678592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4708691386880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4575541678592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250672113994752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49490233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239438418678784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247425602665168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4708504681984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45332106 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244147134969344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65923103 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4575629700608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65912960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16692017 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49490231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28639824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255247743695360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250672096955392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255247743695360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250672113994752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115413336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49490233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239438418678784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247425602665168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4708504681984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45332106 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244147134969344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65923103 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4575629700608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65912960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16692017 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49490231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28639824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255247743695360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250672096955392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255247743695360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250672113994752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115413336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251986859830039 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255247638633984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115403191 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45331841 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239438443582464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 $ _timestamp‚1.784499525611995e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49490233 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255247638633984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65912960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4575629700608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4575541678592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4708691386880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4575541678592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250672113994752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4576219788288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252024511037189 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65933015 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16694663 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115420410 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49497552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6499 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4576219788288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65922858 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49497554 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115430569 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45338780 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244183327682560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49497554 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4562019264901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247462491772288 % _timestamp‚1.7844995258694792e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250709480361984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115420410 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4561933116315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45338541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255285788179456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4709385864192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49497552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255285788179456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252024407849243 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65922858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4709213846528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115430569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4576219788288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252024511037189 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65933015 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16694663 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115420410 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49497552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6499 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4576219788288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65922858 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49497554 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115430569 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45338780 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244183327682560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49497554 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4562019264901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247462491772288 % _timestamp‚1.7844995258694792e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250709480361984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115420410 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4561933116315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45338541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255285788179456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4709385864192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49497552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255285788179456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252024407849243 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65922858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4709213846528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115430569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4743499754496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253864395881281 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚6549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245966676295680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252540415316992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4609479420416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28854324 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4609479420416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252540432356352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45671863 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16817802 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249269223092528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49856281 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45672129 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253864292693447 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257149894737408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66409712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245966883709952 _runtime‚2618.080359681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116276138 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252540415316992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49856283 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚529 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4743682265088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609567442432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257149999798784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66419855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49856281 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28854327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚257149999798784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257149894737408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49856283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 eQDeviceStatsMonitor.on_train_batch_start/inactivd;².Ûe_split_bytes.large_pool.allocated‚241223201444864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116265993 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66419855 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16817539 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252540432356352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249269206053168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680t“0üÝ'Ð×O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 % _timestamp‚1.7844995441671202e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609567442432 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4743499754496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253864395881281 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚6549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245966676295680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252540415316992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4609479420416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28854324 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4609479420416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252540432356352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45671863 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16817802 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249269223092528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49856281 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45672129 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253864292693447 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257149894737408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66409712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245966883709952 _runtime‚2618.080359681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116276138 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252540415316992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49856283 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚529 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4743682265088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609567442432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257149999798784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66419855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49856281 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28854327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚257149999798784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257149894737408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49856283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241223201444864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116265993 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66419855 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16817539 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252540432356352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249269206053168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152‘äÎfU2Ñ- _wandbruntime‚2618  _runtime‚2618ÕV®â&ÒÜM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4595848740309 _step‚530 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116283212 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257187939214336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16820218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4744196073984 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49863602 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2618.336396797 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253901840712651 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116293371 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49863604 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7844995444252825e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66429767 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49863602 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249306095160288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49863604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16819959 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚246003281470976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66419610 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4744384868864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252577798723584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241258871698432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4610157530112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6549 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45678573 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4595762591723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116293371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257188044282880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241258896602112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2618.336396797 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253901840712651 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116293371 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49863604 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7844995444252825e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66429767 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49863602 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249306095160288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49863604 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16819959 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚246003281470976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66419610 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4744384868864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252577798723584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241258871698432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4610157530112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6549 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45678573 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4595762591723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116293371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257188044282880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241258896602112 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257187939214336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16820218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4744196073984 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49863602 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254408750717952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46011982 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 2636.83426522 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259052150840832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50222333 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46011711 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚254408750717952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16942887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4629002264161 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7844995629234438e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243007934403584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251112826480528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29068827 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247786406535168 trainer/global_step‚6599 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4643417162240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117128795 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247786611852288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4628916115687 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255741725556855 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4778652545024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4643417162240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚254408733678592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50222331 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66916607 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50222333 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254408733678592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243007959307264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251112809441168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117138940 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66906464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29068824 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4778472131584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255741828744689 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66906464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16943155 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50222331 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66916607 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259052255902208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259052150840832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4643505184256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4778472131584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255741828744689 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66906464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16943155 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50222331 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66916607 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259052255902208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259052150840832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4643505184256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254408750717952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46011982 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 2636.83426522 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259052150840832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50222333 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46011711•]Ca[2ß- _wandbruntime‚2637  _runtime‚2637’dž‹¼$àÜM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46018499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29073114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247822801524224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117146014 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29073117 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254446100045824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4629592067131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4779171963392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255779273576059 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16945385 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50229652 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50229652 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66926519 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4629678215717 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117156173 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66916362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚6599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117156173 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251149698548288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4644095271936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255779376764005 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259090300386304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66916362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50229654 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7844995631795392e+09 _runtime‚2637.090654766 _step‚534 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259090300386304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259090195317760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254446117085184 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0`óûVç&ááM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46018499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29073114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247822801524224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117146014 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29073117 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254446100045824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4629592067131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4779171963392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255779273576059 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16945385 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50229652 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50229652 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66926519 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4629678215717 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117156173 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66916362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚6599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117156173 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251149698548288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4644095271936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255779376764005 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259090300386304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66916362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50229654 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7844995631795392e+09 _runtime‚2637.090654766 _step‚534 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259090300386304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259090195317760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254446117085184 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4677442926080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50588383 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4813454618112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260954406944256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244792692265984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117991597 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260954406944256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4677354904064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29283327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29283324 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4677442926080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50588383 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4813454618112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260954406944256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244792692265984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117991597 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260954406944256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4677354904064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29283327 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29283324 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50595704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46357865 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17069981 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257656706439467 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50595702 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29287614 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249642749700096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118018975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29287617 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252993301936288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46357595 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4663507691125 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249642540195840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4663421542539 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118018975 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260992556489728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67413114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260992556489728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17070248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244828387423232 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4678121042944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4814337373184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚538 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4678121042944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50595702 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257656809627413 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2—ÌBæ&ñàM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29287617 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252993301936288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46357595 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4663507691125 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249642540195840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4663421542539 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118018975 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260992556489728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67413114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260992556489728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17070248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244828387423232 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4678121042944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4814337373184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚538 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4678121042944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50595702 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257656809627413 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50595704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46357865 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17069981 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257656706439467 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50595702 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29287614 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249642749700096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118018975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482š„¼„K2ò- _wandbruntime‚2655  _runtime‚2655BN¼’óÿ /loss/FBt2/detection_loss‚0.5983673360834301 &loss/total_loss‚10.318642953545428 trainer/global_step‚6649 0loss/FRt2/assignment_loss‚0.9799295051922816 2loss/SRqqt2/assignment_loss‚0.7823222082186962 1loss/FBt2/assignment_loss‚0.15737334664779795 /loss/FBt1/detection_loss‚0.8412314475979656 2loss/FBt1/assignment_loss‚0.032518973419563224 /loss/FRt1/detection_loss‚0.5974777506521192 epoch‚8 % _timestamp‚1.7844995819109774e+09 _runtime‚2655.821578721 _step‚539 /loss/FRt1/assignment_loss‚2.293316857750751 2loss/SRqqt1/assignment_loss‚0.7381208087631072 0loss/SRqqt2/detection_loss‚0.697824268034515 .loss/FRt2/detection_loss‚1.466542021201449 0loss/SRqqt1/detection_loss‚1.133618429983751J  1 ô„ .loss/FRt2/detection_loss‚1.466542021201449 0loss/SRqqt1/detection_loss‚1.133618429983751 /loss/FBt2/detection_loss‚0.5983673360834301 &loss/total_loss‚10.318642953545428 trainer/global_step‚6649 0loss/FRt2/assignment_loss‚0.9799295051922816 2loss/SRqqt2/assignment_loss‚0.7823222082186962 1loss/FBt2/assignment_loss‚0.15737334664779795 /loss/FBt1/detection_loss‚0.8412314475979656 2loss/FBt1/assignment_loss‚0.032518973419563224 /loss/FRt1/detection_loss‚0.5974777506521192 epoch‚8 % _timestamp‚1.7844995819109774e+09 _runtime‚2655.821578721 _step‚539 /loss/FRt1/assignment_loss‚2.293316857750751 2loss/SRqqt1/assignment_loss‚0.7381208087631072 0loss/SRqqt2/detection_loss‚0.697824268034515›o^1\`õ‚(:Õ Œ›õÒÒð+ disk.md0.in‚0" disk.dm-1.out‚1477.65673828125 cpu‚26.310218738980243 network.sent‚ 8.844881e+06 disk.md0.out‚ 77.06640625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 7.37890625& proc.memory.availableMB‚ 107.671875" proc.memory.rssMB‚ 2851.7578125) proc.memory.percent‚17.40574836730957 proc.cpu.threads‚17% disk.nvme0n1.out‚12908.4228515625! disk.dm-0.out‚20575.267578125 disk.rbd0.in‚0 network.recv‚799365' disk.nvme1c1n1.out‚9309.18701171875 disk.nvme1n1.in‚1.625% disk.nvme1n1.out‚9309.18701171875 disk.dm-0.in‚ 9.00390625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚12908.4228515625$ memory_percent‚99.34282302856445) disk./.usagePercent‚76.14949842439648% disk./.usageGB‚1169.0851745605469 disk.nvme0n1.in‚ 7.37890625 disk.nvme1c1n1.in‚1.625^Çî«ö‚(:„ Œ›õÒÇÉœ, gpu.0.gpu‚73.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚195.711' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.237 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07M½P2÷- _wandbruntime‚2674  _runtime‚2674–ÀŸpÇøÁ & lr-AdamW/pg2‚9.619227331241799e-05 trainer/global_step‚6699 % _timestamp‚1.7844996004538538e+09 _runtime‚2674.364763076 _step‚540 & lr-AdamW/pg1‚9.619227331241799e-05†jKÖÌùÆ & lr-AdamW/pg1‚9.619227331241799e-05 & lr-AdamW/pg2‚9.619227331241799e-05 trainer/global_step‚6699 % _timestamp‚1.7844996004538538e+09 _runtime‚2674.364763076 _step‚540œµ¹ñ.2ú- _wandbruntime‚2674  _runtime‚2674é²uˆ+ûÒO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4696661214977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246577475032064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711292645888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251425827027968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118854399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4711380667904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259496694471505 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67910111 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17191746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251426044928000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4696575066503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.smalU§Ž5­l_pool.freed‚ 4848376899584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711292645888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258145370401792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46689839 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17192012 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50954433 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67910111 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262856768109056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259496591283671 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254800033256528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67899968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46689570 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262856663047680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262856663047680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711292645888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251425827027968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118854399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4711380667904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259496694471505 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67910111 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17191746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251426044928000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4696575066503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4848376899584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711292645888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258145370401792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46689839 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17192012 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50954433 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67910111 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262856768109056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259496591283671 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254800033256528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67899968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46689570 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104Nx›42ý- _wandbruntime‚2674  _runtime‚2674Í0s7á&þÛM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4849071646208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4697251017947 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚254836888284928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4711970755584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17194231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118871618 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67909866 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67920023 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118871618 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50961752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50961754 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46696345 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17194485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118881777 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2674.623046696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258182753808384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29502114 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4711970755584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222087168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67909866 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118881777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262894812593152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246613170189312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254836905324288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50961752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262894707524608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258182736769024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712058784768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246613145285632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚542 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67920023 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46696602 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784499600711983e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251462216931840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488*@×Ä@"ÿàM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29502114 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4711970755584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222087168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67909866 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118881777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262894812593152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246613170189312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254836905324288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50961752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262894707524608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258182736769024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712058784768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246613145285632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚542 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67920023 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46696602 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784499600711983e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251462216931840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4849071646208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4697251017947 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚254836888284928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4711970755584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17194231 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118871618 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67909866 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67920023 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118871618 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50961752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50961754 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitorùpÌ"¦.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46696345 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17194485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118881777 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2674.623046696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258182753808384 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152ž)‡2€- _wandbruntime‚2674  _runtime‚2674€ç.Á€ 0loss/FRt2/assignment_loss‚1.0653765859111908 0loss/SRqqt1/detection_loss‚1.050689491216326 1loss/SRqqt2/detection_loss‚0.6513929251851814 _runtime‚2674.623568997 _step‚543 /loss/FRt1/assignment_loss‚2.301436698225279 2loss/SRqqt1/assignment_loss‚0.6511271705676336 1loss/FBt2/assignment_loss‚0.00344357962308095 &loss/total_loss‚10.723017191426754 epoch‚8 /loss/FRt1/detection_loss‚0.6104412656939191 /loss/FRt2/detection_loss‚1.4148356006868805 /loss/FBt2/detection_loss‚1.2202868424714184 trainer/global_step‚6699 2loss/SRqqt2/assignment_loss‚0.6368924153424012 1loss/FBt1/assignment_loss‚0.30622595635732563 /loss/FBt1/detection_loss‚0.8108686601461175 % _timestamp‚1.7844996007129073e+099 Ô1 ‚… 0loss/SRqqt1/detection_loss‚1.050689491216326 1loss/SRqqt2/detection_loss‚0.6513929251851814 _runtime‚2674.623568997 _step‚543 /loss/FRt1/assignment_loss‚2.301436698225279 2loss/SRqqt1/assignment_loss‚0.6511271705676336 1loss/FBt2/assignment_loss‚0.00344357962308095 &loss/total_loss‚10.723017191426754 epoch‚8 /loss/FRt1/detection_loss‚0.6104412656939191 /loss/FRt2/detection_loss‚1.4148356006868805 /loss/FBt2/detection_loss‚1.2202868424714184 trainer/global_step‚6699 2loss/SRqqt2/assignment_loss‚0.6368924153424012 1loss/FBt1/assignment_loss‚0.30622595635732563 /loss/FBt1/detection_loss‚0.8108686601461175 % _timestamp‚1.7844996007129073e+09 0loss/FRt2/assignment_loss‚1.0653765859111908Ÿn³w•šƒ‚(: ››õÒ©æµ+ gpu.0.gpu‚59.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚195.565' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.18833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ío›}]„‚(:Ò ››õÒõ•ü+) proc.memory.percent‚17.40574836730957 network.sent‚ 8.907235e+06) disk./.usagePercent‚76.14950364235072 disk.nvme0n1.in‚ 7.37890625 disk.nvme1n1.in‚1.625$ memory_percent‚99.34325218200684' proc.memory.availableMB‚ 107.6015625 network.recv‚804139 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 7.37890625# disk.nvme0n1.out‚12909.68359375 disk.nvme1c1n1.in‚1.625! disk.dm-0.out‚20575.837890625" proc.memory.rssMB‚ 2851.7578125 cpu‚26.308306792049205 disk.dm-1.in‚0" disk.dm-1.out‚1477.78955078125 disk.rbd0.out‚0% disk.nvme0c0n1.out‚12909.68359375' disk.nvme1c1n1.out‚9310.20556640625 disk.md0.out‚ 77.81640625 proc.cpu.threads‚17% disk./.usageGB‚1169.0852546691895% disk.nvme1n1.out‚9310.20556640625 disk.md0.in‚0 disk.dm-0.in‚ 9.00390625k“¸x2…- _wandbruntime‚2692  _runtime‚2692€-Î'ƆÀ $ _timestamp‚1.784499618960698e+09 _runtime‚2692.871634719 _step‚544 & lr-AdamW/pg1‚9.609931300855686e-05 & lr-AdamW/pg2‚9.609931300855686e-05 trainer/global_step‚67496+]ˇŠ& lr-AdamW/pg1‚9.609931300855686e-05 & lr-AdamW/pg2‚9.609931300855686e-05 trainer/global_step‚6749 $ _timestamp‚1.784499618960698e+09 _runtime‚2692.871634719 _step‚544 égô2ˆ- _wandbruntime‚2692  _runtime‚2692µ{Šo×'‰ÑO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260013688763392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4730404541911 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119717201 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4745318409728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256643636644528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17316253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68396720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17315999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29712324 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253245554540032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚ 2692.87238522 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264758919151104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844996189615805e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51320481 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260013688763392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264759024212480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119727346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29712327 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51320483 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253245757760000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248362232894464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47028323 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4883346549248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚6749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264759024212480 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4745230387712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68406863 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256643619605168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68406863 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4745230387712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261374024147079 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47028580 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4883524865536 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68396720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51320481 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260013705802752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248362207990784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730490690385 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚545 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264758919151104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260013705802752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0%\/ÂÜ'ŠÖO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260013688763392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264759024212480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119727346 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29712327 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51320483 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253245757760000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248362232894464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47028323 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4883346549248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚6749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264759024212480 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4745230387712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68406863 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256643619605168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68406863 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4745230387712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261374024147079 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47028580 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4883524865536 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68396720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51320481 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260013705802752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248362207990784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730490690385 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚545 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264758919151104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260013705802752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260013688763392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4730404541911 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119717201 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4745318409728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256643636644528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17316253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68396720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17315999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29712324 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253245554540032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚ 2692.87238522 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264758919151104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844996189615805e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51320481 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584¡M ^2‹- _wandbruntime‚2693  _runtime‚2693”¸ÖäŒÜM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4884226842624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731080493355 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68416775 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68406618 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68406618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51327804 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚261411675354229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119734420 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119744579 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119734420 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚6749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51327804 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264797068696576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚253282154894336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256680508712288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260051055130624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119744579 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4745908497408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260051055130624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68416775 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17318743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253281945390080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29716617 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47035116 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29716614 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚261411572166283 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745996526592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17318502 NADeviceStatsMonitor.on_tra'ßþin_batch_end/allocation.large_pool.freed‚51327802 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260051072169984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248397903148032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745996526592 _runtime‚2693.130409733 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264797068696576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4884042242048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51327802 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260051072169984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745908497408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248ÜÖÓÐç&áM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745996526592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17318502 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51327802 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260051072169984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248397903148032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745996526592 _runtime‚2693.130409733 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264797068696576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4884042242048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51327802 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260051072169984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745908497408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4884226842624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731080493355 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68416775 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68406618 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68406618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51327804 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚261411675354229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119734420 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119744579 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119734420 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚6749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51327804 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264797068696576 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚253282154894336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256680508712288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260051055130624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119744579 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4745908497408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260051055130624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68416775 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17318743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253281945390080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29716617 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47035116 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29716614 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚261411572166283 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159¢Ñ‚¦2Ž- _wandbruntime‚2693  _runtime‚2693ÿüÏ€ .loss/FRt1/assignment_loss‚2.18268021895932 2loss/SRqqt2/assignment_loss‚0.6792263963518211 1loss/FBt1/assignment_loss‚0.04876559845892617 /loss/FRt1/detection_loss‚0.5735516704921744 1loss/SRqqt2/detection_loss‚0.6624640077181697 2loss/FBt2/assignment_loss‚0.010363659730233375 1loss/SRqqt1/detection_loss‚1.1189669063020073 /loss/FBt2/detection_loss‚0.6181371733625993 %loss/total_loss‚9.990803612492538 epoch‚8 _runtime‚2693.130896234 _step‚547 0loss/FRt2/assignment_loss‚0.8252873787116178 2loss/SRqqt1/assignment_loss‚0.8858443502251271 /loss/FRt2/detection_loss‚1.5229626675841934 /loss/FBt1/detection_loss‚0.8625535845963491 trainer/global_step‚6749 % _timestamp‚1.7844996192202375e+09'"\ … 0loss/FRt2/assignment_loss‚0.8252873787116178 2loss/SRqqt1/assignment_loss‚0.8858443502251271 /loss/FRt2/detection_loss‚1.5229626675841934 /loss/FBt1/detection_loss‚0.8625535845963491 trainer/global_step‚6749 % _timestamp‚1.7844996192202375e+09 .loss/FRt1/assignment_loss‚2.18268021895932 2loss/SRqqt2/assignment_loss‚0.6792263963518211 1loss/FBt1/assignment_loss‚0.04876559845892617 /loss/FRt1/detection_loss‚0.5735516704921744 1loss/SRqqt2/detection_loss‚0.6624640077181697 2loss/FBt2/assignment_loss‚0.010363659730233375 1loss/SRqqt1/detection_loss‚1.1189669063020073 /loss/FBt2/detection_loss‚0.6181371733625993 %loss/total_loss‚9.990803612492538 epoch‚8 _runtime‚2693.130896234 _step‚547£á¾…š‘‚(: ª›õÒÄ›Í+ gpu.0.gpu‚15.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.691' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.89699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é""b’‚(:× ª›õÒ‚Š¿, network.sent‚ 8.956878e+06% disk./.usageGB‚1168.0685920715332 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 7.37890625& disk.nvme0n1.out‚12919.70068359375 disk.md0.out‚ 77.984375 disk.dm-0.in‚ 9.00390625) disk./.usagePercent‚76.08328233652716 disk.nvme1c1n1.in‚1.625% disk.nvme1n1.out‚9323.77880859375 disk.dm-1.in‚0 cpu‚26.306223362555073 proc.cpu.threads‚17 network.recv‚808590 disk.md0.in‚0! disk.dm-1.out‚1477.9052734375( proc.memory.availableMB‚ 107.56640625) proc.memory.percent‚17.40574836730957 disk.nvme0c0n1.in‚ 7.37890625( disk.nvme0c0n1.out‚12919.70068359375' disk.nvme1c1n1.out‚9323.77880859375 disk.nvme1n1.in‚1.625" disk.dm-0.out‚20598.9208984375$ memory_percent‚99.34346675872803" proc.memory.rssMB‚ 2851.75781250T¸2“- _wandbruntime‚2711  _runtime‚2711¢ý8Å”¿ % lr-AdamW/pg2‚9.60056284050165e-05 trainer/global_step‚6799 % _timestamp‚1.7844996377981758e+09 _runtime‚2711.709706881 _step‚548 % lr-AdamW/pg1‚9.60056284050165e-05„Ç9®Ê•Ä _runtime‚2711.709706881 _step‚548 % lr-AdamW/pg1‚9.60056284050165e-05 % lr-AdamW/pg2‚9.60056284050165e-05 trainer/global_step‚6799 % _timestamp‚1.7844996377981758e+09¤›©ÿg2–- _wandbruntime‚2711  _runtime‚2711Á(}8×'—ÑO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120580003 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261882024164352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68893472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29926824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4764234017319 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68903615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47368666 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263251560198321 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚261882024164352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261882007124992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51686531 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51686533 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚2711.709797921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29926827 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4779256151552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 trainer/global_step‚6799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4779168129536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4779256151552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68893472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258487240032528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258487222993168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250146965853184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255065529356800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263251457010487 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68903615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17441570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266661175254528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261882007124992Õ˜ÖO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68893472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29926824 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4764234017319 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68903615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47368666 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263251560198321 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚261882024164352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261882007124992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51686531 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51686533 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚2711.709797921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29926827 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4779256151552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 trainer/global_step‚6799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4779168129536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4779256151552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68893472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258487240032528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258487222993168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_*»ë % batch_start/inactive_split.small_pool.peak‚487 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250146965853184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255065529356800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263251457010487 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68903615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17441570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266661175254528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261882007124992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120580003 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261882024164352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152¥~rÜ2™- _wandbruntime‚2711  _runtime‚2711[—5þâ&šÜM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255101916763136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250182661010432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29931114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17444182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120607381 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68903370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120597222 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120597222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258524095060928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4764996117349 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚550 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47375042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29931117 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚2711.967662499 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266699219731456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51693854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261919373492224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263289108217637 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261919390531584 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68903370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266699324800000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7844996380565727e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4779846239232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51693854 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51693852 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258524112100288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4764909968763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51693852 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261919373492224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266699324800000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4779934268416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263289005029691 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47375299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4919230849024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120607381 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68913527 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4779934268416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4779846239232 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250182661010432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29931114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17444182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120607381 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68903370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120597222 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120597222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258524095060928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4764996117349 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚550 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47375042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29931117 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚2711.967662499 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266699219731456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51693854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261919373492224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263289108217637 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261919390531584 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68903370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266699324800000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7844996380565727e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4779846239232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51693854 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51693852 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258524112100288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4764909968763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51693852 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261919373492224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266699324800000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4779934268416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263289005029691 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47375299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4919230849024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120607381 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68913527 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4779934268416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4779846239232 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268563431357952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121452950 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251931723715584 trainer/global_step‚6849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17566275 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256885234698752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251931748619264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4813193893376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263750325486592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚wy˜s17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30141327 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263750342525952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4813193893376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69390224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52052583 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265128889873895 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7844996565741603e+09 _runtime‚2730.488367557 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52052581 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚553 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4813105871360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256885029381632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268563431357952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121452950 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251931723715584 trainer/global_step‚6849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17566275 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256885234698752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251931748619264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4813193893376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263750325486592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30141327 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263750342525952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4813193893376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69390224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52052583 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265128889873895 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7844996565741603e+09 _runtime‚2730.488367557 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52052581 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚553 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4813105871360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256885029381632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47714398 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4813783981056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121470183 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52059902 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚268601475834880 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚268601580903424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52059904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17568784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251967443776512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4954005602304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚260367715488288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52059904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52059902 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260367698448928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚554 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47714655 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263787691853824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4954188105728 _runtime‚2730.742127817 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121460024­sBy«áM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121460024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30145614 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47714398 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4813783981056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121470183 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52059902 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚268601475834880 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚268601580903424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52059904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17568784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251967443776512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4954005602304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚260367715488288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52059904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52059902 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260367698448928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚554 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47714655 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263787691853824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4954188105728 _runtime‚2730.742127817 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ª –úµ2¬- _wandbruntime‚2730  _runtime‚2730<¢3­‚ &loss/total_loss‚10.138307085383465 epoch‚8 % _timestamp‚1.7844996568320048e+09 _step‚555 0loss/FRt2/assignment_loss‚1.0233530525688652 2loss/SRqqt2/assignment_loss‚0.7441440367744644 1loss/FBt1/assignment_loss‚0.14807590489246086 /loss/FRt1/detection_loss‚0.5473601667093375 _runtime‚2730.746361584 1loss/FBt2/assignment_loss‚0.00206277937579346 /loss/FBt1/detection_loss‚0.5508906796182457 0loss/FRt1/assignment_loss‚2.4187308970106196 2loss/SRqqt1/assignment_loss‚0.6909380726141038 1loss/SRqqt1/detection_loss‚1.0981762284554273 1loss/SRqqt2/detection_loss‚0.7267832961183378 trainer/global_step‚6849 /loss/FRt2/detection_loss‚1.6111489780800972 /loss/FBt2/detection_loss‚0.5766429931657123ÓÍKô ®‡ 2loss/SRqqt2/assignment_loss‚0.7441440367744644 1loss/FBt1/assignment_loss‚0.14807590489246086 /loss/FRt1/detection_loss‚0.5473601667093375 _runtime‚2730.746361584 1loss/FBt2/assignment_loss‚0.00206277937579346 /loss/FBt1/detection_loss‚0.5508906796182457 0loss/FRt1/assignment_loss‚2.4187308970106196 2loss/SRqqt1/assignment_loss‚0.6909380726141038 1loss/SRqqt1/detection_loss‚1.0981762284554273 1loss/SRqqt2/detection_loss‚0.7267832961183378 trainer/global_step‚6849 /loss/FRt2/detection_loss‚1.6111489780800972 /loss/FBt2/detection_loss‚0.5766429931657123 &loss/total_loss‚10.138307085383465 epoch‚8 % _timestamp‚1.7844996568320048e+09 _step‚555 0loss/FRt2/assignment_loss‚1.0233530525688652«,| ^¯‚(:Ó ×›õÒƒ¿Á+ network.sent‚ 9.087295e+06! disk.dm-1.out‚1479.0849609375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 7.37890625& disk.nvme1c1n1.out‚9407.4990234375 disk.nvme1n1.in‚1.625$ disk.nvme1n1.out‚9407.4990234375 disk.md0.in‚0) disk./.usagePercent‚76.06367475531074' disk.nvme0c0n1.out‚13094.3818359375 disk.nvme0n1.in‚ 7.37890625$ memory_percent‚99.34415817260742" proc.memory.rssMB‚ 2851.7578125 cpu‚26.300340094544246 proc.cpu.threads‚17% disk./.usageGB‚1167.7675666809082 disk.dm-1.in‚0 disk.dm-0.in‚ 9.00390625) proc.memory.percent‚17.40574836730957 network.recv‚820715 disk.rbd0.in‚0% disk.nvme0n1.out‚13094.3818359375 disk.nvme1c1n1.in‚1.625 disk.md0.out‚ 79.03515625" disk.dm-0.out‚20853.8662109375& proc.memory.availableMB‚ 107.453125iæÇ0›°‚(: ×›õÒ–¿—, gpu.0.gpu‚77.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚196.403' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.46766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ûÏô2±- _wandbruntime‚2749  _runtime‚2749µµè¯Æ²À & lr-AdamW/pg1‚9.581609231267146e-05 & lr-AdamW/pg2‚9.581609231267146e-05 trainer/global_step‚6899 $ _timestamp‚1.784499675349947e+09 _runtime‚2749.261492504 _step‚556ÌÕÉ Ë³Å & lr-AdamW/pg1‚9.581609231267146e-05 & lr-AdamW/pg2‚9.581609231267146e-05 trainer/global_step‚6899 $ _timestamp‚1.784499675349947e+09 _runtime‚2749.261492504 _step‚556¬§bd+2´- _wandbruntime‚2749  _runtime‚2749/ÒÖ×'µÑO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270465687461376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69886976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚262174446808528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784499675350616e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270465687461376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265618643848192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4831979116609 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚3300Ñ|-Ü'¶ÖO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270465687461376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265618643848192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4831979116609 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270465687461376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69886976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚262174446808528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784499675350616e+09­Óýa2·- _wandbruntime‚2749  _runtime‚2749Ó;¸ÜM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48053422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17693562 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4989141325312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258741342964224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253752201638912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270503731938304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30360117 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚558 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69907031 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265656010215424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265656027254784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253752176735232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262211318876288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4832655068165 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832568919579 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270503837006848 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122332985 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267043870756507 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52425954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69907031 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _runtime‚2749.521273688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4988952530432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265656027254784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847809752064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122322826 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847721722880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258741129265664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17693308 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270503837006848 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122332985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847721722880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362ʤ Žç&¹áM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265656010215424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265656027254784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253752176735232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262211318876288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4832655068165 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832568919579 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270503837006848 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122332985 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267043870756507 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52425954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69907031 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _runtime‚2749.521273688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4988952530432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265656027254784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847809752064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122322826 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847721722880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258741129265664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17693308 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270503837006848 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122332985 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847721722880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48053422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17693562 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4989141325312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258741342964224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253752201638912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270503731938304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30360117 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚558 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69907031®Qá±2º- _wandbruntime‚2749  _runtime‚2749%S1‘» epoch‚8 trainer/global_step‚6899 2loss/FBt2/assignment_loss‚0.008966861187077898 .loss/FRt2/detection_loss‚1.429530306187598 1loss/SRqqt1/detection_loss‚1.1893554067527745 _runtime‚2749.521751939 _step‚559 0loss/FRt2/assignment_loss‚0.9767518564757065 2loss/SRqqt2/assignment_loss‚0.7362409674802776 0loss/FBt1/assignment_loss‚0.3448718336339809 &loss/total_loss‚10.533662972987274 % _timestamp‚1.7844996756111166e+09 0loss/FRt1/assignment_loss‚2.4562075299731756 2loss/SRqqt1/assignment_loss‚0.8187493562901681 /loss/FRt1/detection_loss‚0.5345870295399343 1loss/SRqqt2/detection_loss‚0.5735590120528558 /loss/FBt1/detection_loss‚0.9261781012173742 /loss/FBt2/detection_loss‚0.5386647121963507ÔuÎ ¼† /loss/FBt2/detection_loss‚0.5386647121963507 epoch‚8 trainer/global_step‚6899 2loss/FBt2/assignment_loss‚0.008966861187077898 .loss/FRt2/detection_loss‚1.429530306187598 1loss/SRqqt1/detection_loss‚1.1893554067527745 _runtime‚2749.521751939 _step‚559 0loss/FRt2/assignment_loss‚0.9767518564757065 2loss/SRqqt2/assignment_loss‚0.7362409674802776 0loss/FBt1/assignment_loss‚0.3448718336339809 &loss/total_loss‚10.533662972987274 % _timestamp‚1.7844996756111166e+09 0loss/FRt1/assignment_loss‚2.4562075299731756 2loss/SRqqt1/assignment_loss‚0.8187493562901681 /loss/FRt1/detection_loss‚0.5345870295399343 1loss/SRqqt2/detection_loss‚0.5735590120528558 /loss/FBt1/detection_loss‚0.9261781012173742¯°iZ‚R½‚(:Ç æ›õÒúìô+ network.sent‚ 9.162379e+06# disk.nvme1n1.out‚9534.330078125# disk.dm-0.out‚20856.56103515625 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 13097.265625% disk.nvme1c1n1.out‚9534.330078125# memory_percent‚99.3443489074707 cpu‚26.298527193838577 network.recv‚824684' disk./.usagePercent‚76.063680221739$ disk./.usageGB‚1167.767650604248 disk.md0.in‚0 disk.md0.out‚ 79.41796875 disk.dm-0.in‚ 9.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚1.625" disk.dm-1.out‚1605.26611328125 disk.nvme0n1.in‚ 7.37890625! disk.nvme0n1.out‚ 13097.265625 disk.nvme1c1n1.in‚1.625& proc.memory.availableMB‚ 107.421875" proc.memory.rssMB‚ 2851.7578125) proc.memory.percent‚17.40574836730957 disk.nvme0c0n1.in‚ 7.37890625 proc.cpu.threads‚17ùŸ<+›¾‚(: æ›õÒ¯È, gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚198.616' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.20533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R}¨è2¿- _wandbruntime‚2767  _runtime‚2767êˆÈìÆÀÀ & lr-AdamW/pg1‚9.572024385967198e-05 & lr-AdamW/pg2‚9.572024385967198e-05 trainer/global_step‚6949 $ _timestamp‚1.784499693822212e+09 _runtime‚2767.735381379 _step‚560‡À‘ËÁÅ _runtime‚2767.735381379 _step‚560 & lr-AdamW/pg1‚9.572024385967198e-05 & lr-AdamW/pg2‚9.572024385967198e-05 trainer/global_step‚6949 $ _timestamp‚1.784499693822212e+09°!9ë2Â- _wandbruntime‚2767  _runtime‚2767šw:×'ÃÑO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17814934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272367943564800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267486962209792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268883755600711 trainer/global_step‚6949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267486979249152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _step‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123178554 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4865808592017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52784683 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255501239440384 _runtime‚2767.735438889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267486962209792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197190656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70393871 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264018050196528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4880981355008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272368048626176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52784681 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264018033157168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272368048626176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30570324 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268883858788545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260524406407168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4880981355008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5023166966784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30570327 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4881069377024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5023364157440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4881069377024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48385532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17815205 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4865722443543 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272367943564800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267486979249152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70393871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222094336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70383728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784499693822784e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52784683 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52784681 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255501264344064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48385258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260524628501504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 œË ôÄÖO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52784681 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264018033157168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272368048626176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30570324 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268883858788545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260524406407168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4880981355008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5023166966784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30570327 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4881069377024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5023364157440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4881069377024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48385532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17815205 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4865722443543 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272367943564800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267486979249152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70393871 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222094336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70383728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784499693822784e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52784683 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52784681 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255501264344064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48385258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260524628501504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17814934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272367943564800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267486962209792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268883755600711 trainer/global_step‚6949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267486979249152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _step‚561 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123178554 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4865808592017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52784683 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255501239440384 _runtime‚2767.735438889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267486962209792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197190656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70393871 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264018050196528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4880981355008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272368048626176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330±jDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚562 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52792002 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264054905224928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17817601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4866398394987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30574617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260561004684800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123185628 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52792002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48392218 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264054922264288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 trainer/global_step‚6949 _runtime‚2768.299107722 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4866484543573 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267524345616384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4881747493888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17817347 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844996943880498e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268921303619915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267524328577024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48391961 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123195787 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5023871068672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482ïWPç&ÇáM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70403783 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272405988041728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268921406807861 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264054922264288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 trainer/global_step‚6949 _runtime‚2768.299107722 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4866484543573 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267524345616384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4881747493888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17817347 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844996943880498e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268921303619915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267524328577024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48391961 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123195787 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5023871068672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267524345616384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4881747493888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70393626 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255536959501312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5024045183488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚562 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52792002 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264054905224928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17817601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4866398394987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30574617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260561004684800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123185628 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52792002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48392218 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 Fké2Ð- _wandbruntime‚2786  _runtime‚2786?ä·‹ÑÑO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4899551918951 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257285997302784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262344363052032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4915007118848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274270304729600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53150731 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257286022206464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53150731 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70890623 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262344161929216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5058340845568 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265861653584528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269355280571392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4914919096832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269355280571392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274270304729600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269355297610752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265861636545168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚2786.532372749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30784824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70880480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17940259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4899638067425 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4914919096832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53150733 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124041356 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4915007118848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274270199668224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274270199668224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269355297610752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17940493 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5058164626432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270761188464119 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48725320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ž‘¹[Ü'ÒÖO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274270304729600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53150731 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257286022206464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53150731 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70890623 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262344161929216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5058340845568 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265861653584528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269355280571392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4914919096832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269355280571392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274270304729600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269355297610752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265861636545168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚2786.532372749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30784824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70880480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17940259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4899638067425 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4914919096832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53150733 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124041356 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4915007118848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274270199668224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274270199668224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269355297610752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17940493 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5058164626432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270761188464119 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48725320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4899551918951 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257285997302784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262344363052032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4915007118848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143µ¡Iò£2Ó- _wandbruntime‚2786  _runtime‚2786g–e£á&ÔÛM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270798839671269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70890378 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48731866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4900314018981 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274308349213696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4915597206528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274308349213696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274308244145152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53158052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30789117 _runtime‚2786.788270429 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124048430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262380558603776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265898508612928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265898525652288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30789114 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257321692460032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 trainer/global_step‚6999 $ _timestamp‚1.784499712877097e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4900227870395 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124048430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269392663977984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269392646938624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124058589 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17943003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚566é –¢ÕàM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30789114 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257321692460032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 trainer/global_step‚6999 $ _timestamp‚1.784499712877097e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4900227870395 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124048430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269392663977984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269392646938624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124058589 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17943003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚566 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270798839671269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70890378 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48731866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4900314018981 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274308349213696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4915597206528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274308349213696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274308244145152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53158052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30789117 _runtime‚2786.788270429 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124048430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262380558603776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265898508612928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265898525652288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¶=ÐY|2Ö- _wandbruntime‚2786  _runtime‚2786ž,Yׂ /loss/FRt2/detection_loss‚1.3742502861614547 _step‚567 2loss/SRqqt2/assignment_loss‚0.5643154613352683 2loss/FBt2/assignment_loss‚0.002482072775328561 /loss/FRt1/detection_loss‚0.6241109596747846 1loss/SRqqt2/detection_loss‚0.7211047619100407 0loss/FRt1/assignment_loss‚2.2492037869941597 0loss/FRt2/assignment_loss‚0.9587668616380265 .loss/FBt1/detection_loss‚0.968995496954607 &loss/total_loss‚10.061705323719355 epoch‚8 % _timestamp‚1.7844997128780897e+09 2loss/SRqqt1/assignment_loss‚0.7808393256211443 1loss/FBt1/assignment_loss‚0.10715038840819192 1loss/SRqqt1/detection_loss‚1.0926840101670943 /loss/FBt2/detection_loss‚0.6178019120792548 trainer/global_step‚6999 _runtime‚2786.788776249Å𓇠؇ epoch‚8 % _timestamp‚1.7844997128780897e+09 2loss/SRqqt1/assignment_loss‚0.7808393256211443 1loss/FBt1/assignment_loss‚0.10715038840819192 1loss/SRqqt1/detection_loss‚1.0926840101670943 /loss/FBt2/detection_loss‚0.6178019120792548 trainer/global_step‚6999 _runtime‚2786.788776249 /loss/FRt2/detection_loss‚1.3742502861614547 _step‚567 2loss/SRqqt2/assignment_loss‚0.5643154613352683 2loss/FBt2/assignment_loss‚0.002482072775328561 /loss/FRt1/detection_loss‚0.6241109596747846 1loss/SRqqt2/detection_loss‚0.7211047619100407 0loss/FRt1/assignment_loss‚2.2492037869941597 0loss/FRt2/assignment_loss‚0.9587668616380265 .loss/FBt1/detection_loss‚0.968995496954607 &loss/total_loss‚10.061705323719355·†;+ZÙ‚(:Ï „œõÒ‚“¯+$ memory_percent‚99.34499263763428 cpu‚26.294807867340033" disk.dm-1.out‚1949.33349609375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 7.37890625# disk.nvme0c0n1.out‚ 13100.265625 disk.nvme0n1.in‚ 7.37890625! disk.nvme0n1.out‚ 13100.265625 proc.cpu.threads‚17 network.recv‚833366) disk./.usagePercent‚76.06368345190116 disk.dm-1.in‚0& disk.nvme1c1n1.out‚9879.6005859375$ disk.nvme1n1.out‚9879.6005859375 disk.md0.in‚0# disk.dm-0.out‚20858.87353515625( proc.memory.availableMB‚ 107.31640625 disk.nvme1c1n1.in‚1.625 network.sent‚ 9.273071e+06% disk./.usageGB‚1167.7677001953125 disk.nvme1n1.in‚1.625 disk.dm-0.in‚ 9.00390625" proc.memory.rssMB‚ 2851.7578125) proc.memory.percent‚17.40574836730957 disk.rbd0.in‚0 disk.md0.out‚ 80.31640625¨ŠÆÚ‚(:„ „œõÒò؆, gpu.0.gpu‚69.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚202.152' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª’mF`Û‚(:Õ “œõÒ Ã+ disk.nvme1n1.in‚1.625 disk.rbd0.in‚0$ memory_percent‚99.34501647949219 network.sent‚ 9.337366e+06 disk.nvme0c0n1.in‚ 7.37890625 disk.nvme0n1.in‚ 7.37890625' disk.nvme1c1n1.out‚9904.50048828125 disk.md0.in‚0 disk.dm-0.in‚ 9.00390625 disk.dm-1.in‚0" disk.dm-1.out‚1953.29833984375% disk./.usageGB‚1168.0365371704102( disk.nvme0c0n1.out‚13213.23388671875 disk.nvme1c1n1.in‚1.625% disk.nvme1n1.out‚9904.50048828125$ proc.memory.availableMB‚107.3125) proc.memory.percent‚17.40574836730957 proc.cpu.threads‚17( disk./.usagePercent‚76.0811944094063 disk.md0.out‚ 80.9921875# disk.dm-0.out‚20991.33642578125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2851.7578125 cpu‚26.292792016287702 network.recv‚837869& disk.nvme0n1.out‚13213.23388671875DîïóÜ‚(:„ “œõÒä¶ì+ gpu.0.gpu‚46.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.963' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.321 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ j›2Ý- _wandbruntime‚2805  _runtime‚2805PAôÇÞÁ % _timestamp‚1.7844997314630768e+09 _runtime‚2805.374006972 _step‚568 & lr-AdamW/pg1‚9.552639381603225e-05 & lr-AdamW/pg2‚9.552639381603225e-05 trainer/global_step‚70492þš\ÌßÆ % _timestamp‚1.7844997314630768e+09 _runtime‚2805.374006972 _step‚568 & lr-AdamW/pg1‚9.552639381603225e-05 & lr-AdamW/pg2‚9.552639381603225e-05 trainer/global_step‚7049¸`œ——2à- _wandbruntime‚2805  _runtime‚2805 %3×'áÑO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚276172560833024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267705256972528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4933467542833 _runtime‚2805.374732353 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276172560833024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53516783 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4948944860672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49063935 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49063683 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4948856838656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18064608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚569 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124894013 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7049 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259070780068864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53516781 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5093120170496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 $ _timestamp‚1.784499731463929e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124904158 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18064359 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267705239933168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30999327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276172455771648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271223615972352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4948944860672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264164078555648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5093298486784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276172455771648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272638724515361 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53516781 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71377232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272638621327527 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271223615972352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71377232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4948856838656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30999324 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4933381394359 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276172560833024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53516783 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4948944860672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49063935 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49063683 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4948856838656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18064608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚569 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124894013 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7049 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259070780068864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53516781 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5093120170496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 $ _timestamp‚1.784499731463929e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124904158 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18064359 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267705239933168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30999327 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276172455771648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271223615972352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4948944860672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264164078555648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5093298486784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276172455771648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272638724515361 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53516781 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71377232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272638621327527 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271223615972352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71377232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4948856838656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30999324 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4933381394359 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚570 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267742112000928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4934057345803 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124921391 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271260965300224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4949534948352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267742129040288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276210605317120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264200265354240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53524104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71397287 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49070479 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5093815031808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18067109 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272676169346731 % _timestamp‚1.7844997317236392e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276210500248576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/activv”Þ‘e.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271260982339584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31003617 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5093997535232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259106475226112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2805.634707065 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124921391 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71387130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272676272534677 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4949622977536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53524102 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4949534948352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚7049 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271260965300224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4949622977536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71387130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53524102 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124911232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104¥ í«ç&åáM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276210500248576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271260982339584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31003617 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5093997535232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259106475226112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2805.634707065 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124921391 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71387130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272676272534677 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4949622977536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53524102 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4949534948352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚7049 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271260965300224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4949622977536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71387130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53524102 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124911232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53524104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259106450322432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚570 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267742112000928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4934057345803 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124921391 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271260965300224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4949534948352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267742129040288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276210605317120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264200265354240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53524104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71397287 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49070479 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5093815031808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18067109 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272676169346731 % _timestamp‚1.7844997317236392e+09ºˆr¥2æ- _wandbruntime‚2805  _runtime‚2805ŒAiç 1loss/FBt2/assignment_loss‚0.11387689222324915 /loss/FRt1/detection_loss‚0.6290478122229918 1loss/SRqqt1/detection_loss‚0.9831564235240684 trainer/global_step‚7049 _step‚571 0loss/FRt1/assignment_loss‚2.2857391191025576 1loss/FBt1/assignment_loss‚0.07099456395112921 /loss/FBt2/detection_loss‚0.6832235691507563 2loss/SRqqt1/assignment_loss‚0.6222911806956443 /loss/FRt2/detection_loss‚1.3583953453360924 1loss/SRqqt2/detection_loss‚0.6728114639253666 epoch‚8 % _timestamp‚1.7844997317245593e+09 _runtime‚2805.635181326 0loss/FRt2/assignment_loss‚1.0198030278086663 2loss/SRqqt2/assignment_loss‚0.6477773757172852 /loss/FBt1/detection_loss‚1.8606900748881428 %loss/total_loss‚10.94780684854595ŸÁÈB è† 0loss/FRt2/assignment_loss‚1.0198030278086663 2loss/SRqqt2/assignment_loss‚0.6477773757172852 /loss/FBt1/detection_loss‚1.8606900748881428 %loss/total_loss‚10.94780684854595 1loss/FBt2/assignment_loss‚0.11387689222324915 /loss/FRt1/detection_loss‚0.6290478122229918 1loss/SRqqt1/detection_loss‚0.9831564235240684 trainer/global_step‚7049 _step‚571 0loss/FRt1/assignment_loss‚2.2857391191025576 1loss/FBt1/assignment_loss‚0.07099456395112921 /loss/FBt2/detection_loss‚0.6832235691507563 2loss/SRqqt1/assignment_loss‚0.6222911806956443 /loss/FRt2/detection_loss‚1.3583953453360924 1loss/SRqqt2/detection_loss‚0.6728114639253666 epoch‚8 % _timestamp‚1.7844997317245593e+09 _runtime‚2805.635181326»­É7^é‚(:Ó ¢œõÒ—âÅ+) disk./.usagePercent‚76.07918326075507" disk.dm-0.out‚21107.5439453125 disk.nvme0c0n1.in‚ 7.37890625 disk.nvme1c1n1.in‚1.625& disk.nvme1n1.out‚10446.85009765625 disk.md0.in‚0% disk./.usageGB‚1168.0056610107422 disk.dm-0.in‚ 9.00390625" disk.dm-1.out‚2489.12353515625 disk.nvme0n1.in‚ 7.37890625% disk.nvme0n1.out‚13323.5595703125 disk.md0.out‚81.28125# memory_percent‚99.3466854095459 network.sent‚ 9.340288e+06 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚13323.5595703125( disk.nvme1c1n1.out‚10446.85009765625 disk.nvme1n1.in‚1.625' proc.memory.availableMB‚ 107.0390625) proc.memory.percent‚17.40574836730957 network.recv‚841094 disk.rbd0.out‚0" proc.memory.rssMB‚ 2851.7578125 cpu‚26.2910649420005 proc.cpu.threads‚17j~‘÷ê‚(:„ ¢œõÒ¹¿—, gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚115.845' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.615 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%î{¸2ë- _wandbruntime‚2824  _runtime‚2824•§ÇìÁ _step‚572 & lr-AdamW/pg1‚9.542839533029389e-05 & lr-AdamW/pg2‚9.542839533029389e-05 trainer/global_step‚7099 % _timestamp‚1.7844997503277926e+09 _runtime‚2824.238721708dV#äÌíÆ & lr-AdamW/pg1‚9.542839533029389e-05 & lr-AdamW/pg2‚9.542839533029389e-05 trainer/global_step‚7099 % _timestamp‚1.7844997503277926e+09 _runtime‚2824.238721708 _step‚572¼¹wÐg2î- _wandbruntime‚2824  _runtime‚2824.ÂÀŸØ'ïÒO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚273091934333952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278074711875072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71884127 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71873984 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265983821972992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53882833 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚573 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4967297018241 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53882833 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5128101531136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269548843321168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49402455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4967210869767 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278074711875072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269548860360528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚278074816936448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982794580480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274516157378769 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53882831 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71873984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚273091917294592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125756815 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278074816936448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18188628 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265983614558720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚273091934333952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 trainer/global_step‚7099S h§, ð×O LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125756815 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278074816936448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18188628 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265983614558720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚273091934333952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 trainer/global_step‚7099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚273091934333952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278074711875072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71884127 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71873984 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265983821972992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53882833 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚573 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4967297018241 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53882833 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5128101531136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269548843321168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49402455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4967210869767 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278074711875072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269548860360528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚278074816936448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982794580480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274516157378769 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53882831 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71873984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚273091917294592½!âD2ñ- _wandbruntime‚2824  _runtime‚2824gÿ /â&òÜM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4983472690176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273129300701184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31218114 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5128804911104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4967972969797 _step‚574 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71883882 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚2824.495920784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4983560719360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125784193 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269585732428288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125774034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266020218405888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844997505848281e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278112861420544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273129300701184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71883882 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278112861420544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274553705398085 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4983472690176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49408981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4967886821211 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53890152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18191119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18190867 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53890154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274553602210139 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53890152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18191119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18190867 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53890154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274553602210139 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273129300701184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31218114 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5128804911104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4967972969797 _step‚574 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71883882 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚2824.495920784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4983560719360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125784193 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269585732428288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125774034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266020218405888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844997505848281e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278112861420544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273129300701184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71883882 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278112861420544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274553705398085 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4983472690176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49408981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4967886821211¾½žI›2ô- _wandbruntime‚2824  _runtime‚2824á+H£õ‚ /loss/FRt1/detection_loss‚0.6290504001652407 /loss/FRt2/detection_loss‚1.4656993527896702 1loss/SRqqt1/detection_loss‚1.0701374913896284 /loss/SRqqt2/detection_loss‚0.61115790816697 0loss/FBt2/detection_loss‚0.48965971177334294 epoch‚8 _runtime‚2824.496371205 2loss/SRqqt1/assignment_loss‚0.6888725834042435 2loss/FBt1/assignment_loss‚0.027093927807697697 %loss/total_loss‚9.999678375203082 /loss/FBt1/detection_loss‚1.4305707930456957 _step‚575 0loss/FRt2/assignment_loss‚0.8261063714748819 2loss/SRqqt2/assignment_loss‚0.5515551763738007 2loss/FBt2/assignment_loss‚0.005223757902105236 trainer/global_step‚7099 % _timestamp‚1.7844997505857613e+09 0loss/FRt1/assignment_loss‚2.2045509009098043K ö‡ /loss/FRt2/detection_loss‚1.4656993527896702 1loss/SRqqt1/detection_loss‚1.0701374913896284 /loss/SRqqt2/detection_loss‚0.61115790816697 0loss/FBt2/detection_loss‚0.48965971177334294 epoch‚8 _runtime‚2824.496371205 2loss/SRqqt1/assignment_loss‚0.6888725834042435 2loss/FBt1/assignment_loss‚0.027093927807697697 %loss/total_loss‚9.999678375203082 /loss/FBt1/detection_loss‚1.4305707930456957 _step‚575 0loss/FRt2/assignment_loss‚0.8261063714748819 2loss/SRqqt2/assignment_loss‚0.5515551763738007 2loss/FBt2/assignment_loss‚0.005223757902105236 trainer/global_step‚7099 % _timestamp‚1.7844997505857613e+09 0loss/FRt1/assignment_loss‚2.2045509009098043 /loss/FRt1/detection_loss‚0.6290504001652407¿¼·¹oN÷‚(:à ±œõÒñŠõ+ network.recv‚844689 disk.nvme0n1.out‚ 13332.875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 7.43359375) proc.memory.percent‚17.40574836730957 network.sent‚ 9.414461e+06 disk.nvme1c1n1.in‚1.625 disk.nvme1n1.in‚1.625 disk.md0.in‚0 disk.dm-0.in‚ 9.05859375 disk.rbd0.out‚0$ memory_percent‚99.34539794921875" proc.memory.availableMB‚107.25) disk./.usagePercent‚76.08381879191856% disk./.usageGB‚1168.0768280029297( disk.nvme1c1n1.out‚10472.60009765625& disk.nvme1n1.out‚10472.60009765625 disk.md0.out‚ 83.5390625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2851.7578125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7.43359375 disk.dm-0.out‚21119.96484375" disk.dm-1.out‚2507.15869140625 disk.nvme0c0n1.out‚ 13332.875 cpu‚26.292070404404665Ô?"áø‚(:„ ±œõÒ•ù¡, gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚201.318' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈbE2ù- _wandbruntime‚2844  _runtime‚2844“àÚîÇúÁ trainer/global_step‚7149 % _timestamp‚1.7844997702807367e+09 _runtime‚2844.192395453 _step‚576 & lr-AdamW/pg1‚9.532968329098467e-05 & lr-AdamW/pg2‚9.532968329098467e-05((”aÌûÆ _step‚576 & lr-AdamW/pg1‚9.532968329098467e-05 & lr-AdamW/pg2‚9.532968329098467e-05 trainer/global_step‚7149 % _timestamp‚1.7844997702807367e+09 _runtime‚2844.192395453ÀãQÉš2ü- _wandbruntime‚2844  _runtime‚2844÷pA½× ýÑO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31428324 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5016732322304 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279976967978496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31428327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18312045 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126619617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚262640295793664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279977073039872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5016820344320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 $ _timestamp‚1.784499770281505e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5163240484352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _MDeviceStatsMÐ8<onitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5163057973760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274960252695552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274960235656192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54248883 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72380879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 _runtime‚2844.192469093 _step‚577 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72380879 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274960235656192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5016732322304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72370736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126629762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49740372 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267803536278016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274960252695552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5001040345175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54248883 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276393590242177 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262640270889984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279976967978496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31428327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18312045 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126619617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚262640295793664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279977073039872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5016820344320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 $ _timestamp‚1.784499770281505e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5163240484352 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5163057973760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274960252695552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274960235656192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54248883 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72380879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 _runtime‚2844.192469093 _step‚577 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72380879 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274960235656192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5016732322304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72370736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126629762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49740372 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267803536278016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274960252695552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5001040345175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54248883 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276393590242177 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262640270889984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5001802445205 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54256204 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54256204 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5017498461184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚262675966047232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267839931155456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262675990950912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276431035073547 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271429318776928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126636836 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5163757701120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274997602023424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49746802 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274997602023424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18314463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72380634 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72380634 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280015012455424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5017410432000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276431138261493 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126646995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5001716296619 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54256202 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280015117523968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280015117523968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49747080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 1266469955V1ãïßM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5001802445205 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54256204 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54256204 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5017498461184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚262675966047232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267839931155456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262675990950912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276431035073547 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271429318776928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126636836 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5163757701120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274997602023424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49746802 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274997602023424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18314463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72380634 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72380634 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280015012455424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5017410432000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276431138261493 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126646995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5001716296619 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54256202 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280015117523968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280015117523968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49747080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126646995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B›ˆ‚(: ÏœõÒ€¨ò+ gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚165.233' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚55.07766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}UœÁ2‰- _wandbruntime‚2865  _runtime‚2865ñà£zÇŠÁ % _timestamp‚1.7844997912189045e+09 _runtime‚2865.129848495 _step‚580 & lr-AdamW/pg1‚9.523025927917831e-05 & lr-AdamW/pg2‚9.523025927917831e-05 trainer/global_step‚7199qì²Ì‹Æ _runtime‚2865.129848495 _step‚580 & lr-AdamW/pg1‚9.523025927917831e-05 & lr-AdamW/pg2‚9.523025927917831e-05 trainer/global_step‚7199 % _timestamp‚1.7844997912189045e+09ÄáÌ72Œ- _wandbruntime‚2865  _runtime‚2865wtc×'ÑO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127492564 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18435343 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264425028752384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264425053656064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72877631 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281879329143296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5034955969057 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269623238736384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54614933 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198185080320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5050670064128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72877631 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚2865.130735237 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276828571057152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 $ _timestamp‚1.784499791219903e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5198004666880 trainer/global_step‚7199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5050758086144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281879224081920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50078407 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5050670064128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273236050097168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54614931 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127482419 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281879224081920 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54614933 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5050758086144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281879329143296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278271023105585 _step‚581 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276828554017792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273236067136528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54614931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31642827 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31642824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18435580 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278270919917751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72867488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722múl^Ü'ŽÖO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269623238736384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54614933 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198185080320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5050670064128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72877631 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚2865.130735237 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276828571057152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 $ _timestamp‚1.784499791219903e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5198004666880 trainer/global_step‚7199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5050758086144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281879224081920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50078407 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5050670064128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273236050097168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54614931 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127482419 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281879224081920 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54614933 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5050758086144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281879329143296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278271023105585 _step‚581 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276828554017792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273236067136528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54614931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31642827 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31642824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18435580 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278270919917751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72867488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127492564 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18435343 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264425028752384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264425053656064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72877631 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281879329143296 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5034955969057Å•»®T2- _wandbruntime‚2865  _runtime‚2865É\(äz ÚM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5035631920613 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚2865.405109311 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54622254 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72877386 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127499638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264460748813312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281917373627392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264460723909632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5198696493056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269659629906944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72887543 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276865920385024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54622252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273272939204288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5051436203008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50084855 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278308467936955 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31647117 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18437741 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚7199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5051348173824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269659420402688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54622252 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5051348173824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127509797 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50085096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152ÇK­*å&‘ßM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281917373627392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264460723909632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5198696493056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269659629906944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72887543 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276865920385024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54622252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273272939204288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5051436203008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50084855 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278308467936955 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31647117 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18437741 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚7199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5051348173824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269659420402688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54622252 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5051348173824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127509797 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50085096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5035631920613 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚2865.405109311 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54622254 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72877386 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127499638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264460748813312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448Æj“]²2’- _wandbruntime‚2865  _runtime‚2865ñẓ _step‚583 1loss/SRqqt2/detection_loss‚0.7757476838094568 _runtime‚2865.405772372 2loss/SRqqt1/assignment_loss‚0.5839863794091611 2loss/SRqqt2/assignment_loss‚0.7759526389034809 3loss/FBt2/assignment_loss‚0.0023160208128834413 1loss/SRqqt1/detection_loss‚1.0294330726162746 .loss/FBt1/detection_loss‚0.879808280163926 /loss/FRt1/assignment_loss‚2.261376879487164 1loss/FBt1/assignment_loss‚0.03701499920826461 /loss/FRt1/detection_loss‚0.6058101625485932 /loss/FRt2/detection_loss‚1.5188612341460652 /loss/FBt2/detection_loss‚0.4832522372659574 epoch‚8 trainer/global_step‚7199 $ _timestamp‚1.784499791495099e+09 0loss/FRt2/assignment_loss‚1.1303810972468298 &loss/total_loss‚10.083940685618057A| e ”† 2loss/SRqqt1/assignment_loss‚0.5839863794091611 2loss/SRqqt2/assignment_loss‚0.7759526389034809 3loss/FBt2/assignment_loss‚0.0023160208128834413 1loss/SRqqt1/detection_loss‚1.0294330726162746 .loss/FBt1/detection_loss‚0.879808280163926 /loss/FRt1/assignment_loss‚2.261376879487164 1loss/FBt1/assignment_loss‚0.03701499920826461 /loss/FRt1/detection_loss‚0.6058101625485932 /loss/FRt2/detection_loss‚1.5188612341460652 /loss/FBt2/detection_loss‚0.4832522372659574 epoch‚8 trainer/global_step‚7199 $ _timestamp‚1.784499791495099e+09 0loss/FRt2/assignment_loss‚1.1303810972468298 &loss/total_loss‚10.083940685618057 _step‚583 1loss/SRqqt2/detection_loss‚0.7757476838094568 _runtime‚2865.405772372Çщ$ic•‚(:Ø ÞœõÒ»Š¬+( disk.nvme1c1n1.out‚10484.03271484375 proc.cpu.threads‚17) disk./.usagePercent‚76.25297418422603 disk.md0.in‚0 disk.md0.out‚ 85.640625' disk.nvme0c0n1.out‚16085.0498046875 disk.nvme0n1.in‚ 14.71484375$ memory_percent‚99.38302040100098# proc.memory.rssMB‚ 2857.69921875* proc.memory.percent‚17.442011833190918 disk.dm-1.in‚0" disk.dm-1.out‚2511.03662109375 disk.nvme1c1n1.in‚3.25 cpu‚26.317869137280713 network.recv‚856977# disk./.usageGB‚1170.67378616333 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 17.96484375" disk.dm-0.out‚23875.3349609375' proc.memory.availableMB‚ 101.0859375 network.sent‚ 9.532007e+06& disk.nvme1n1.out‚10484.03271484375 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 14.71484375% disk.nvme0n1.out‚16085.0498046875¨œ"b–‚(:„ ÞœõÒÏÛ‚, gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚145.398' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚48.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È+º 2—- _wandbruntime‚2885  _runtime‚2885õŒX 0˜‹{ *jet/accuracy_1_of_3‚0.9223922491073608 *jet/accuracy_1_of_4‚0.9652193784713745 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7403664588928223 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7523030042648315 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02963908016681671 +jet/accuracy_3_of_4‚0.09701317548751831 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039667557924985886 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6518527269363403 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29998138546943665 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.639032244682312 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04251052066683769 *jet/accuracy_2_of_4‚0.8235071897506714 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1403062641620636 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.718787670135498 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.56910240650177 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8104349970817566 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7158119082450867 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 .particle/accuracy_3_of_3‚0.999514102935791 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9939153790473938 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5161806344985962 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 +jet/accuracy_3_of_3‚0.06755231320858002 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 )loss/val_total_loss‚9.587166230810302 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8394197821617126 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3832577168941498 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17197586596012115 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7629704475402832 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7912279367446899 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7249889373779297 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4621756374835968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.045302052050828934 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17138920724391937 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_1_of_1‚0.31716209650039673 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17714445292949677 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6713770031929016 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939526915550232 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8088101148605347 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17932771146297455 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5015063285827637 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9594488739967346 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9979138374328613 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.40220847725868225 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9572198987007141 particle/accuracy_1_of_1‚1 +particle/specificity‚0.9126121997833252 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2630535066127777 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3833945691585541 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6417468786239624 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9795634150505066 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2671734392642975 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4563894271850586 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4128788113594055 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6009132862091064 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920229315757751 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988896250724792 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024901488795876503 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12338733673095703 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7373568415641785 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 jet/accuracy_2_of_5‚1 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.484527051448822 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927017688751221 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19371779263019562 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18146422505378723 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20437994599342346 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04331227019429207 jet/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1746177226305008 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01994219981133938 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02963908016681671 /particle/accuracy_3_of_4‚0.9981684684753418 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04000396654009819 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6906391382217407 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.643743634223938 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5043342709541321 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7867792844772339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7632074952125549 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 _runtime‚2885.992450972 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6602795720100403 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6002200245857239 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6980530023574829 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9769479036331177 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 )jet/accuracy_1_of_2‚0.781999945640564 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.991209864616394 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14905545115470886 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚8 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7282105088233948 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6837002038955688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6406679749488831 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5208333730697632 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9786099791526794 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18146422505378723 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12700653076171875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21816349029541016 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7807016968727112 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24805273115634918 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6532845497131348 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03843989595770836 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8466821312904358 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6499346494674683 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9942424297332764 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8989668488502502 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37962958216667175 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972943663597107 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03548048064112663 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_4_of_4‚0.01071428507566452 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19241568446159363 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9008263349533081 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13949234783649445 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15035289525985718 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5456844568252563 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6336508393287659 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15804417431354523 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6911215782165527 9"Purity/*FRt*SRqqt1FBt/SRq]ÇN„ qt_purity‚0.5542724132537842 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8951814770698547 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012476983480155468 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15580376982688904 +jet/accuracy_2_of_2‚0.14650215208530426 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5425471067428589 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8527133464813232 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20053628087043762 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6validation_average_jet_accuracy‚0.3617135286331177 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5937500596046448 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9819602966308594 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 'particle/f_score‚0.6487845778465271 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 _step‚584 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15920273959636688 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297726631164551 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44366368651390076 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7458308935165405 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19387145340442657 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11059621721506119 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7579721808433533 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983579516410828 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5980554819107056 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8643699884414673 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20054744184017181 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8051280975341797 jet/accuracy_4_of_5‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9918051362037659 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7429930567741394 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚7208 $ _timestamp‚1.784499812081106e+09 *jet/accuracy_2_of_3‚0.5573594570159912 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7954516410827637 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6271412968635559 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1672419309616089 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 jet/accuracy_1_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8643699884414673 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6499346494674683 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46890759468078613 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475H`Ÿ–=™{ :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01994219981133938 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02963908016681671 /particle/accuracy_3_of_4‚0.9981684684753418 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04000396654009819 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6906391382217407 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.643743634223938 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5043342709541321 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7867792844772339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7632074952125549 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 _runtime‚2885.992450972 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6602795720100403 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6002200245857239 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6980530023574829 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9769479036331177 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 )jet/accuracy_1_of_2‚0.781999945640564 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.991209864616394 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14905545115470886 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚8 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7282105088233948 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6837002038955688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6406679749488831 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5208333730697632 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9786099791526794 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18146422505378723 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12700653076171875 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21816349029541016 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7807016968727112 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24805273115634918 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6532845497131348 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03843989595770836 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8466821312904358 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6499346494674683 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9942424297332764 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8989668488502502 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37962958216667175 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972943663597107 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03548048064112663 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_4_of_4‚0.01071428507566452 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19241568446159363 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9008263349533081 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13949234783649445 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15035289525985718 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5456844568252563 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6336508393287659 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15804417431354523 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6911215782165527 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5542724132537842 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8951814770698547 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012476983480155468 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15580376982688904 +jet/accuracy_2_of_2‚0.14650215208530426 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5425471067428589 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8527133464813232 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20053628087043762 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6validation_average_jet_accuracy‚0.3617135286331177 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5937500596046448 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9819602966308594 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 'particle/f_score‚0.6487845778465271 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 _step‚584 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15920273959636688 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297726631164551 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44366368651390076 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7458308935165405 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19387145340442657 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11059621721506119 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7579721808433533 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983579516410828 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5980554819107056 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8643699884414673 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20054744184017181 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8051280975341797 jet/accuracy_4_of_5‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9918051362037659 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7429930567741394 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚7208 $ _timestamp‚1.784499812081106e+09 *jet/accuracy_2_of_3‚0.5573594570159912 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7954516410827637 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6271412968635559 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1672419309616089 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 jet/accuracy_1_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8643699884414673 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6499346494674683 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46890759468078613 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 *jet/accuracy_1_of_3‚0.9223922491073608 *jet/accuracy_1_of_4‚0.9652193784713745 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7403664588928223 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7523030042648315 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02963908016681671 +jet/accuracy_3_of_4‚0.09701317548751831 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039667557924985886 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6518527269363403 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29998138546943665 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.639032244682312 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04251052066683769 *jet/accuracy_2_of_4‚0.8235071897506714 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1403062641620636 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.718787670135498 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.56910240650177 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8104349970817566 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7158119082450867 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 .particle/accuracy_3_of_3‚0.999514102935791 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9939153790473938 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5161806344985962 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 +jet/accuracy_3_of_3‚0.06755231320858002 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 )loss/val_total_loss‚9.587166230810302 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8394197821617126 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3832577168941498 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17197586596012115 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7629704475402832 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7912279367446899 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7249889373779297 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4621756374835968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.045302052050828934 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17138920724391937 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_1_of_1‚0.31716209650039673 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17714445292949677 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6713770031929016 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939526915550232 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8088101148605347 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17932771146297455 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5015063285827637 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9594488739967346 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9979138374328613 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.40220847725868225 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9572198987007141 particle/accuracy_1_of_1‚1 +particle/specificity‚0.9126121997833252 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2630535066127777 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3833945691585541 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6417468786239624 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9795634150505066 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2671734392642975 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4563894271850586 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4128788113594055 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6009132862091064 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920229315757751 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988896250724792 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024901488795876503 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12338733673095703 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7373568415641785 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 jet/accuracy_2_of_5‚1 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.484527051448822 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927017688751221 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19371779263019562 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18146422505378723 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20437994599342346 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04331227019429207 jet/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1746177226305008È0.Seš‚(:Ú íœõÒœ”³+ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚10601.90869140625& proc.memory.availableMB‚ 192.859375) disk./.usagePercent‚76.36121319086112 disk.md0.in‚0 disk.md0.out‚ 86.28515625( disk.nvme0c0n1.out‚18173.42529296875& disk.nvme0n1.out‚18173.42529296875$ memory_percent‚98.82287979125977 proc.memory.rssMB‚ 2867.4375* proc.memory.percent‚17.501449584960938 network.recv‚861452% disk./.usageGB‚1172.3355255126953 disk.nvme1n1.in‚3.25 network.sent‚ 9.594335e+06 disk.dm-0.in‚ 18.01171875! disk.nvme0c0n1.in‚ 14.76171875 disk.nvme0n1.in‚ 14.76171875& disk.nvme1n1.out‚10601.90869140625 cpu‚26.322074262247284 proc.cpu.threads‚17# disk.dm-0.out‚25990.85791015625 disk.dm-1.in‚0" disk.dm-1.out‚2600.39599609375òÓ]'š›‚(: íœõÒýƇ, gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚188.779' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.92633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%O2œ- _wandbruntime‚2902  _runtime‚2902¦wäÇÁ & lr-AdamW/pg1‚9.513012488735212e-05 & lr-AdamW/pg2‚9.513012488735212e-05 trainer/global_step‚7249 % _timestamp‚1.7844998290192368e+09 _runtime‚2902.930055244 _step‚585]› wÌžÆ trainer/global_step‚7249 % _timestamp‚1.7844998290192368e+09 _runtime‚2902.930055244 _step‚585 & lr-AdamW/pg1‚9.513012488735212e-05 & lr-AdamW/pg2‚9.513012488735212e-05Éoø¼2Ÿ- _wandbruntime‚2902  _runtime‚2902G 8LØ' ÒO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31951404 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73541649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284774669158912 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279669535974400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284774669158912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276048159992880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5105028123136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5105116145152 trainer/global_step‚7249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5253641787904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50590637 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281137236257077 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281137339444911 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73531506 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128749955 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55218451 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55218449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚586 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5105116145152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73531506 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18639458 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279669553013760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18639233 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284774564097536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50590865 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272388667058688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5089179452031 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279669553013760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267135209878528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5105028123136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.sma‚¡ "ll_pool.allocated‚ 5105116145152 trainer/global_step‚7249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5253641787904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50590637 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281137236257077 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281137339444911 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73531506 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128749955 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55218451 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55218449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚586 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5105116145152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73531506 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18639458 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279669553013760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18639233 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284774564097536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50590865 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272388667058688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5089179452031 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279669553013760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267135209878528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31951404Êûe!2¢- _wandbruntime‚2903  _runtime‚2903Ì¢8Bá&£ÛM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267170905035776 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128767174 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18641667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272425060997120 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50597361 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279706919380992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5105794262016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 trainer/global_step‚7249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284812608574464 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128767174 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272425264209920 _step‚587 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279706902341632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5254334270464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279706919380992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55225772 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128777333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31955697 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18641910 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5105706232832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73541404 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784499829291386e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279706902341632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 â?æ&¤àM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚281174887464227 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281174784276281 _runtime‚2903.202481715 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚284812713643008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31955694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50597361 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279706919380992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5105794262016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 trainer/global_step‚7249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284812608574464 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128767174 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272425264209920 _step‚587 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279706902341632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5254334270464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279706919380992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55225772 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128777333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31955697 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18641910 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5105706232832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73541404 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784499829291386e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279706902341632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267170905035776 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128767174 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18641667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272425060997120 Ed2­- _wandbruntime‚2922  _runtime‚2922 Žr®ÑO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283014669120485 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5288430077952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286676820200960 trainer/global_step‚7299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74038401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74038401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32165904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5122922778965 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50928831 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5122922778965 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50928831 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283014669120485 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5288430077952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286676820200960 trainer/global_step‚7299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74038401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74038401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32165904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50935646 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32170197 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5123684878995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5123598730409 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283052217139689 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277928635448640 % _timestamp‚1.7844998491846986e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74048313 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129640135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18765452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129640135 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5139643974656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5139732003840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74038156 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55591820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281575220703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚281575237742592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5139732003840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286714864677888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55591822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277928618409280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74048313 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚2923.095788616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286714969746432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55591822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5289312017920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129629976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268955687801856 trainer/global_step‚7299 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129629976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268955687801856 trainer/global_step‚7299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50935646 VIDeviceSta÷£wl_tsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32170197 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5123684878995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5123598730409 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283052217139689 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277928635448640 % _timestamp‚1.7844998491846986e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74048313 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129640135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18765452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129640135 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5139643974656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5139732003840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74038156 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55591820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281575220703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚281575237742592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5139732003840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286714864677888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55591822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277928618409280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74048313 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚2923.095788616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286714969746432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55591822 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104Ïjõ4Ú2³- _wandbruntime‚2923  _runtime‚2923û0gŽ ´ƒ /loss/FRt2/detection_loss‚1.3988997498410267 1loss/SRqqt1/detection_loss‚1.3726522063260795 trainer/global_step‚7299 _step‚592 0loss/FRt2/assignment_loss‚0.9337263877920526 1loss/FBt1/assignment_loss‚0.05590360079845474 1loss/SRqqt2/detection_loss‚0.6618731446244218 /loss/FBt1/detection_loss‚1.0264759603887796 &loss/total_loss‚10.247808192017828 0loss/FRt1/assignment_loss‚2.1974685816785655 2loss/SRqqt2/assignment_loss‚0.6183990241412816 3loss/FBt2/assignment_loss‚0.0012962972963934088 epoch‚9 % _timestamp‚1.7844998491857288e+09 2loss/SRqqt1/assignment_loss‚0.6526949623980978 /loss/FRt1/detection_loss‚0.5904077923015397 .loss/FBt2/detection_loss‚0.738010484431135 _runtime‚2923.096366177ïµÚXµˆ /loss/FRt2/detection_loss‚1.3988997498410267 1loss/SRqqt1/detection_loss‚1.3726522063260795 trainer/global_step‚7299 _step‚592 0loss/FRt2/assignment_loss‚0.9337263877920526 1loss/FBt1/assignment_loss‚0.05590360079845474 1loss/SRqqt2/detection_loss‚0.6618731446244218 /loss/FBt1/detection_loss‚1.0264759603887796 &loss/total_loss‚10.247808192017828 0loss/FRt1/assignment_loss‚2.1974685816785655 2loss/SRqqt2/assignment_loss‚0.6183990241412816 3loss/FBt2/assignment_loss‚0.0012962972963934088 epoch‚9 % _timestamp‚1.7844998491857288e+09 2loss/SRqqt1/assignment_loss‚0.6526949623980978 /loss/FRt1/detection_loss‚0.5904077923015397 .loss/FBt2/detection_loss‚0.738010484431135 _runtime‚2923.096366177ÐùJ55Z¶‚(:Ï ‹õÒÅÅÇ+ disk.md0.in‚0$ disk.nvme1n1.out‚10802.552734375 disk.md0.out‚87.34375 disk.dm-0.in‚ 18.01171875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚19057.966796875 disk.nvme0n1.in‚ 14.76171875( proc.memory.availableMB‚ 128.13671875 proc.memory.rssMB‚ 2867.4375) disk./.usagePercent‚76.19528348817438% disk./.usageGB‚1169.7880897521973 disk.nvme1n1.in‚3.25! disk.nvme0c0n1.in‚ 14.76171875$ disk.nvme0n1.out‚19057.966796875$ memory_percent‚99.21791553497314 cpu‚26.317931787298317 proc.cpu.threads‚17 network.sent‚ 9.712423e+06 disk.nvme1c1n1.in‚3.25& disk.nvme1c1n1.out‚10802.552734375" disk.dm-0.out‚27009.2490234375 disk.dm-1.in‚0! disk.dm-1.out‚2664.9443359375 disk.rbd0.in‚0* proc.memory.percent‚17.501449584960938 network.recv‚869956KSKðš·‚(: ‹õÒŸ§—, gpu.0.gpu‚55.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚190.703' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.56766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß´5§Y¸‚(:Î šõÒ‘Þª+" disk.dm-1.out‚2665.68505859375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.38671875$ memory_percent‚99.21798706054688 network.sent‚ 9.774663e+06) disk./.usagePercent‚76.23356513374502% disk./.usageGB‚1170.3758087158203$ disk.nvme0n1.out‚19088.326171875 disk.nvme1c1n1.in‚3.25$ disk.nvme1n1.out‚10807.166015625# disk.dm-0.out‚27042.91552734375& disk.nvme0c0n1.out‚19088.326171875 network.recv‚874038 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 19.63671875# proc.memory.availableMB‚128.125 proc.memory.rssMB‚ 2867.4375* proc.memory.percent‚17.501449584960938 cpu‚26.320045990125045 proc.cpu.threads‚17 disk.nvme0n1.in‚ 16.38671875& disk.nvme1c1n1.out‚10807.166015625 disk.md0.in‚0 disk.md0.out‚ 87.6015625 disk.dm-1.in‚0 disk.rbd0.out‚0:cX™¹‚(:Ž šõÒ¡çç+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.319' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.10633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ŒF2º- _wandbruntime‚2942  _runtime‚2942•Ëv•Ç»Á & lr-AdamW/pg1‚9.492773139041496e-05 & lr-AdamW/pg2‚9.492773139041496e-05 trainer/global_step‚7349 % _timestamp‚1.7844998686869152e+09 _runtime‚2942.598322415 _step‚593âIŸ?Ì¼Æ & lr-AdamW/pg1‚9.492773139041496e-05 & lr-AdamW/pg2‚9.492773139041496e-05 trainer/global_step‚7349 % _timestamp‚1.7844998686869152e+09 _runtime‚2942.598322415 _step‚593Ñ ßæ\2½- _wandbruntime‚2942  _runtime‚2942IŽj×'¾ÑO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130475559 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5172991628800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5172903606784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288579076304384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288579181365760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚288579076304384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279735366768880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74535153 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18887553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276028113998336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55950551 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283406189736960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32380404 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283406189736960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74525010 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55950549 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276028317218304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74535153 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284892101983893 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284892205171727 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 $ _timestamp‚1.784499868687613e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130485704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 _runtime‚2942.598486085 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283406172697600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5156752254373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5156838402847 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283406172697600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279735349729520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152.X#2Ü'¿ÖO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283406189736960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74525010 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55950549 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276028317218304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74535153 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284892101983893 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284892205171727 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 $ _timestamp‚1.784499868687613e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130485704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 _runtime‚2942.598486085 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283406172697600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5156752254373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5156838402847 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283406172697600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279735349729520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130475559 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5172991628800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5172903606784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288579076304384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288579181365760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚288579076304384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279735366768880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74535153 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18887553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276028113998336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55950551 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283406189736960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32380404 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330Ò2w_2À- _wandbruntime‚2942  _runtime‚2942Q–î&ÁÜM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283443539064832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5173581716480 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end¯ÇDB¼$/inactive_split_bytes.all.current‚ 211601408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279772221797280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288617120781312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283443556104192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74545065 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284929650003097 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32384694 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130492778 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51275073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18890122 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130502937 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5173669745664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚595 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74534908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5157514354403 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130502937 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284929753191043 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55957870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74545065 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288617225849856 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844998689591682e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130492778 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5157428205817 trainer/global_step‚7349 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚2942.870414511 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51274816—1ªeç&ÂáM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5157514354403 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130502937 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284929753191043 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55957870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74545065 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288617225849856 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844998689591682e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130492778 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5157428205817 trainer/global_step‚7349 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚2942.870414511 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51274816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283443539064832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5173581716480 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279772221797280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288617120781312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283443556104192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74545065 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284929650003097 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32384694 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130492778 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51275073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18890122 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130502937 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5173669745664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚595 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74534908Ó Eî2Ã- _wandbruntime‚2942  _runtime‚2942Ó´'Äý epoch‚9 2loss/SRqqt2/assignment_loss‚0.6637788230441363 1loss/FBt1/assignment_loss‚0.08167087518641578 0loss/SRqqt1/detection_loss‚1.060386371719975 % _timestamp‚1.7844998689607873e+09 _step‚596 1loss/SRqqt1/assignment_loss‚0.844398849704185 1loss/SRqqt2/detection_loss‚0.6850198809319819 /loss/FBt2/detection_loss‚0.5727693023866103 _runtime‚2942.871501689 /loss/FRt1/assignment_loss‚2.442527921645196 /loss/FRt1/detection_loss‚0.5989455200466461 .loss/FRt2/detection_loss‚1.447268205511916 /loss/FBt1/detection_loss‚0.8447945747863268 &loss/total_loss‚10.391198543086842 /loss/FRt2/assignment_loss‚1.144479200953529 1loss/FBt2/assignment_loss‚0.00515901716992432 trainer/global_step‚7349®¬ÁÅ‚ /loss/FBt1/detection_loss‚0.8447945747863268 &loss/total_loss‚10.391198543086842 /loss/FRt2/assignment_loss‚1.144479200953529 1loss/FBt2/assignment_loss‚0.00515901716992432 trainer/global_step‚7349 epoch‚9 2loss/SRqqt2/assignment_loss‚0.6637788230441363 1loss/FBt1/assignment_loss‚0.08167087518641578 0loss/SRqqt1/detection_loss‚1.060386371719975 % _timestamp‚1.7844998689607873e+09 _step‚596 1loss/SRqqt1/assignment_loss‚0.844398849704185 1loss/SRqqt2/detection_loss‚0.6850198809319819 /loss/FBt2/detection_loss‚0.5727693023866103 _runtime‚2942.871501689 /loss/FRt1/assignment_loss‚2.442527921645196 /loss/FRt1/detection_loss‚0.5989455200466461 .loss/FRt2/detection_loss‚1.447268205511916Ô~ëRbÆ‚(:× ©õÒ»áØ+( disk.nvme1c1n1.out‚10865.81884765625& disk.nvme1n1.out‚10865.81884765625" disk.dm-1.out‚2665.77099609375& disk.nvme0c0n1.out‚19211.208984375 disk.nvme0n1.in‚ 16.38671875 network.sent‚ 9.779507e+06 network.recv‚877249 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.md0.out‚ 87.91015625 disk.dm-0.in‚ 19.63671875( proc.memory.availableMB‚ 128.06640625% disk./.usageGB‚1170.3236122131348" disk.dm-0.out‚27223.6845703125 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.38671875$ disk.nvme0n1.out‚19211.208984375$ memory_percent‚99.21834468841553 proc.memory.rssMB‚ 2867.4375 disk.md0.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.501449584960938 cpu‚26.324570552425048 proc.cpu.threads‚17) disk./.usagePercent‚76.23016526384203çï—ƚǂ(: ©õÒ§õ«, gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.758' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.25266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N7:2È- _wandbruntime‚2963  _runtime‚2963…ÍE&ÇÉÁ & lr-AdamW/pg2‚9.482547552704663e-05 trainer/global_step‚7399 % _timestamp‚1.7844998894260027e+09 _runtime‚2963.337329241 _step‚597 & lr-AdamW/pg1‚9.482547552704663e-05‡.gzÌÊÆ & lr-AdamW/pg2‚9.482547552704663e-05 trainer/global_step‚7399 % _timestamp‚1.7844998894260027e+09 _runtime‚2963.337329241 _step‚597 & lr-AdamW/pg1‚9.482547552704663e-05Õ:@›p2Ë- _wandbruntime‚2963  _runtime‚2963¤ÆXÙ&ÌÒO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5206929370624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290481437469184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75021762 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5206841348608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51606091 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚2963.337569302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32594904 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290481332407808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5358553374720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56316601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19011419 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285274491059200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5358381349888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277847864815616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56316601 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131338361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286769534847301 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290481437469184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272489483465728 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286769638035135 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5206841348608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚281578953117520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56316599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56316599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75031905 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281578970156880 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272489508369408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5190667878255 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19011187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75021762 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 GADeviceStatsMʪJÛÿonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5190581729781 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚7399‹ ¹Ý'Í×O ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285274508098560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290481332407808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5358553374720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56316601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19011419 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285274491059200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5358381349888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277847864815616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56316601 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131338361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286769534847301 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290481437469184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272489483465728 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286769638035135 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5206841348608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚281578953117520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56316599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56316599 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75031905 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281578970156880 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272489508369408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5190667878255 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19011187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75021762 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5190581729781 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚7399 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5206929370624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290481437469184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75021762 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5206841348608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51606091 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚2963.337569302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32594904 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5207519458304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290519376884736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56323920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56323922 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75041817 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5359076641280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75031660 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19013577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277884255264256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131365739 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75031660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19013823 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290519481953280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286807082866505 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272525178622976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277884462671360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32599197 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272525203526656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281615842224640 _runtime‚2963.609613431 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5359259144704 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51612771 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131365739 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56323922 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131355580 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290519481953280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5191343829811 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285311874465792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152#)ùeç&ÐáM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286807186054451 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285311857426432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5207519458304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290519376884736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56323920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56323922 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75041817 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5359076641280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75031660 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19013577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277884255264256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131365739 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75031660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19013823 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290519481953280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286807082866505 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272525178622976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277884462671360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32599197 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272525203526656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281615842224640 _runtime‚2963.609613431 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5359259144704 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51612771 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131365739 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56323922 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131355580 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290519481953280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5191343829811 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285311874465792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152×>©Œ2Ñ- _wandbruntime‚2963  _runtime‚2963íå>Ò‚ 2loss/SRqqt1/assignment_loss‚0.7394780598145612 .loss/FRt2/detection_loss‚1.413125494886259 /loss/FRt1/assignment_loss‚2.344218363914084 2loss/SRqqt2/assignment_loss‚0.5838724921649605 3loss/FBt2/assignment_loss‚0.0027923895549200597 /loss/FRt1/detection_loss‚0.5581260817271578 1loss/SRqqt1/detection_loss‚1.0521854366620187 /loss/FBt1/detection_loss‚0.5943665529944395 %loss/total_loss‚9.415721140420747 0loss/FRt2/assignment_loss‚0.9677820057230812 1loss/FBt1/assignment_loss‚0.10398401549663187 1loss/SRqqt2/detection_loss‚0.6468508158047233 0loss/FBt2/detection_loss‚0.40893943167790986 _runtime‚2963.610414863 _step‚600 epoch‚9 trainer/global_step‚7399 % _timestamp‚1.7844998896995647e+09¶_ Ó‡ 2loss/SRqqt1/assignment_loss‚0.7394780598145612 .loss/FRt2/detection_loss‚1.413125494886259 /loss/FRt1/assignment_loss‚2.344218363914084 2loss/SRqqt2/assignment_loss‚0.5838724921649605 3loss/FBt2/assignment_loss‚0.0027923895549200597 /loss/FRt1/detection_loss‚0.5581260817271578 1loss/SRqqt1/detection_loss‚1.0521854366620187 /loss/FBt1/detection_loss‚0.5943665529944395 %loss/total_loss‚9.415721140420747 0loss/FRt2/assignment_loss‚0.9677820057230812 1loss/FBt1/assignment_loss‚0.10398401549663187 1loss/SRqqt2/detection_loss‚0.6468508158047233 0loss/FBt2/detection_loss‚0.40893943167790986 _runtime‚2963.610414863 _step‚600 epoch‚9 trainer/global_step‚7399 % _timestamp‚1.7844998896995647e+09Ø\¬Vý£Ô‚(:Ì ¸õÒÖßÇ+% disk./.usageGB‚1170.1562118530273 disk.dm-0.out‚27423.16015625# disk.nvme0n1.out‚19290.04296875 disk.nvme1c1n1.in‚3.25$ memory_percent‚99.21872615814209* proc.memory.percent‚17.501449584960938 network.sent‚ 9.841833e+06 network.recv‚881370 disk.nvme1n1.in‚3.25 disk.md0.out‚88.34375! disk.dm-1.out‚2666.5556640625 disk.nvme0n1.in‚ 16.38671875( proc.memory.availableMB‚ 128.00390625 proc.memory.rssMB‚ 2867.4375) disk./.usagePercent‚76.21926147878374 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.38671875' disk.nvme1c1n1.out‚10988.1650390625 proc.cpu.thread”Ž|´s‚17% disk.nvme1n1.out‚10988.1650390625 disk.md0.in‚0 disk.dm-0.in‚ 19.63671875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚19290.04296875 cpu‚26.32796930465135\4— Õ‚(:„ ¸õҥ˚, gpu.0.gpu‚70.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚115.665' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûI$Û2Ö- _wandbruntime‚2983  _runtime‚29838…H8Ç×Á & lr-AdamW/pg2‚9.472251576709188e-05 trainer/global_step‚7449 % _timestamp‚1.7844999098454957e+09 _runtime‚2983.759476813 _step‚601 & lr-AdamW/pg1‚9.472251576709188e-050»a—ÌØÆ trainer/global_step‚7449 % _timestamp‚1.7844999098454957e+09 _runtime‚2983.759476813 _step‚601 & lr-AdamW/pg1‚9.472251576709188e-05 & lr-AdamW/pg2‚9.472251576709188e-05Ùå©`2Ù- _wandbruntime‚2983  _runtime‚2983£PsÿØ'ÚÒO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292383693572608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚2983.760171544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56682649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75518514 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132211308 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5393563006464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287142809420800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287142809420800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75528657 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279667617629696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚602 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5240867112448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292383588511232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32809404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19135335 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279667829238272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51944742 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287142826460160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288646967710709 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5240779090432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274266231808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292383588511232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283422556505520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224497353663 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7844999098493547e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292383693572608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51944491 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274241328128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56682651 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283422573544880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5240867112448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75528657 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5240779090432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132201163 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75518514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5393376301568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152;¡1Ý'Û×O JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75528657 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279667617629696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚602 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5240867112448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292383588511232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32809404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19135335 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279667829238272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51944742 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287142826460160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288646967710709 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5240779090432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274266231808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292383588511232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283422556505520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224497353663 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7844999098493547e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292383693572608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51944491 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274241328128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56682651 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283422573544880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5240867112448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75528657 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5240779090432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132201163 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75518514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5393376301568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292383693572608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚2983.760171544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56682649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75518514 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132211308 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5393563006464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287142809420800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287142809420800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448Ú²O"À2Ü- _wandbruntime‚2984  _runtime‚2984ùŠÿ^â&ÝÜM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132218382 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51951246 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75528412 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288684618917859 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75538569 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5241457200128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292421738056704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32813697 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5241545229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132228541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚7449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19137801 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5394260367360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75538569 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51951498 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283459428573280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚603 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292421632988160 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844999101077402e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288684515729913 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292421738056704 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56689972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132218382 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56689970 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5241545229312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5394073669632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225087156633 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283459445612640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132228541 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287180175788032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287180192827392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75528412 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279704221756416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225173305219 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274309961389056 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚2984.018835512 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56689972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287180192827392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5241457200128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32813694æ]–°ÞáM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56689972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132218382 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56689970 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5241545229312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5394073669632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225087156633 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283459445612640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5Devic¥<å’7"eStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132228541 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287180175788032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287180192827392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75528412 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279704221756416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225173305219 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274309961389056 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚2984.018835512 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56689972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287180192827392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5241457200128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32813694 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132218382 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51951246 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75528412 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288684618917859 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75538569 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5241457200128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292421738056704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32813697 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5241545229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132228541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚7449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19137801 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5394260367360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75538569 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51951498 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283459428573280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚603 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292421632988160 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844999101077402e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288684515729913 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292421738056704 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712ÛÆ8»Š2ß- _wandbruntime‚2984  _runtime‚2984â\'à 1loss/FBt1/assignment_loss‚0.05994619318792047 /loss/FRt1/detection_loss‚0.5367616707791094 1loss/SRqqt1/detection_loss‚0.9640673798012237 /loss/FBt2/detection_loss‚0.7117754683535572 % _timestamp‚1.7844999101092942e+09 2loss/FBt2/assignment_loss‚0.022196707294979332 1loss/SRqqt2/detection_loss‚0.6088612136718016 .loss/FBt1/detection_loss‚9.405681228622417 _step‚604 /loss/FRt1/assignment_loss‚2.265442375829463 &loss/total_loss‚18.260203730304777 0loss/FRt2/assignment_loss‚0.9524238721600601 2loss/SRqqt1/assignment_loss‚0.7849964059547831 2loss/SRqqt2/assignment_loss‚0.5131511159394672 /loss/FRt2/detection_loss‚1.4349000987099967 epoch‚9 trainer/global_step‚7449 _runtime‚2984.019928944Çþ°> ᆠ0loss/FRt2/assignment_loss‚0.9524238721600601 2loss/SRqqt1/assignment_loss‚0.7849964059547831 2loss/SRqqt2/assignment_loss‚0.5131511159394672 /loss/FRt2/detection_loss‚1.4349000987099967 epoch‚9 trainer/global_step‚7449 _runtime‚2984.019928944 1loss/FBt1/assignment_loss‚0.05994619318792047 /loss/FRt1/detection_loss‚0.5367616707791094 1loss/SRqqt1/detection_loss‚0.9640673798012237 /loss/FBt2/detection_loss‚0.7117754683535572 % _timestamp‚1.7844999101092942e+09 2loss/FBt2/assignment_loss‚0.022196707294979332 1loss/SRqqt2/detection_loss‚0.6088612136718016 .loss/FBt1/detection_loss‚9.405681228622417 _step‚604 /loss/FRt1/assignment_loss‚2.265442375829463 &loss/total_loss‚18.260203730304777ܾ ÁUâ‚(:Ê ÇõÒ÷îØ+% disk./.usageGB‚1170.1573219299316# disk.nvme1c1n1.out‚ 10988.890625( proc.memory.availableMB‚ 127.92578125* proc.memory.percent‚17.501449584960938 cpu‚26.330078311197465 network.sent‚ 9.916958e+06) disk./.usagePercent‚76.21933378472117 disk.nvme1n1.in‚3.25 disk.md0.out‚ 88.8515625! disk.dm-1.out‚2666.6533203125 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.38671875 network.recv‚885776 disk.nvme1c1n1.in‚3.25 disk.md0.in‚0 disk.dm-0.in‚ 19.63671875 disk.dm-1.in‚0# memory_percent‚99.2192029953003 proc.memory.rssMB‚ 2867.4375 proc.cpu.threads‚17! disk.nvme1n1.out‚ 10988.890625 disk.dm-0.out‚27425.69921875' disk.nvme0c0n1.out‚19293.0302734375 disk.nvme0n1.in‚ 16.38671875% disk.nvme0n1.out‚19293.0302734375î©‚›ã‚(: ÇõÒ“ö‡, gpu.0.gpu‚59.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.573' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.190999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ€Epaä‚(:Ö ÖõÒå´È+# disk.dm-0.out‚27504.36865234375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚19313.720703125 disk.nvme1n1.in‚3.25& disk.nvme1n1.out‚11048.12255859375# memory_percent‚99.2192029953003( proc.memory.availableMB‚ 127.92578125 proc.memory.rssMB‚ 2867.4375 cpu‚26.333664648359726 proc.cpu.threads‚17 network.sent‚ 9.966402e+06 disk.dm-1.in‚0! disk.dm-1.out‚2666.8486328125! disk.nvme0c0n1.in‚ 16.38671875$ disk.nvme0n1.out‚19313.720703125 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚11048.12255859375* proc.memory.percent‚17.501449584960938) disk./.usagePercent‚76.23919878501376 disk.md0.in‚0 disk.dm-0.in‚ 19.63671875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 16.38671875 network.recv‚889898% disk./.usageGB‚1170.4622993469238 disk.md0.out‚ 89.33984375¸ÄcYŽå‚(:ƒ ÖõÒ¹è‰, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.612' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½\ËÆ2æ- _wandbruntime‚3003  _runtime‚30038äVÉÄç¾ & lr-AdamW/pg2‚9.461885375966027e-05 trainer/global_step‚7499 # _timestamp‚1.78449992914857e+09 _runtime‚ 3003.05961591 _step‚605 & lr-AdamW/pg1‚9.461885375966027e-05Ä[$ñÉèà & lr-AdamW/pg1‚9.461885375966027e-05 & lr-AdamW/pg2‚9.461885375966027e-05 trainer/global_step‚7499 # _timestamp‚1.78449992914857e+09 _runtime‚ 3003.05961591 _step‚605ÝXðF}2é- _wandbruntime‚3003  _runtime‚3003Ÿ®,öØ'êÒO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133074110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290524503761951 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76015266 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33023907 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294285844614656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5274716832256 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281487510819328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133063965 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290524400574117 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57048701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281487307599360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚3003.060340991 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289011144821760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5428486725120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5258326829071 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285266176932880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57048699 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276059024094208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289011144821760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276058999190528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5274804854272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57048701 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57048699 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76025409 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5428308408832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚606 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294285949676032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289011127782400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5274804854272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294285949676032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5274716832256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289011144821760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276058999190528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5274804854272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57048701 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57048699 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76025409 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5428308408832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚606 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294285949676032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289011127782400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5274804854272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294285949676032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5274716832256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294285844614656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5274716832256 JDeviceStatsM2§+qconitor.on_train_batch_start/active.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281487510819328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133063965 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290524400574117 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57048701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281487307599360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚3003.060340991 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289011144821760ÞÄií¢2ì- _wandbruntime‚3003  _runtime‚3003`[Dâ&íÜM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294323994160128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290562051781267 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3003.317365771 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5275394941952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57056022 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281523700218880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57056022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57056020 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57056020 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76035321 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76025164 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52288161 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133091343 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285303031961280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294323889091584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5275482971136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276094719251456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290561948593321 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289048494149632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5275482971136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52288405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33028194 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76025164 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289048511188992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133081184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚294323994160128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289048511188992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33028197 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133091343 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276094694347776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57056022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57056020 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57056020 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76035321 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76025164 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52288161 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133091343 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285303031961280 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294323889091584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5275482971136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276094719251456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290561948593321 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289048494149632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5275482971136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52288405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33028194 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76025164 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289048511188992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133081184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚294323994160128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289048511188992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33028197 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133091343 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276094694347776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19381237 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290879463183360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287109763281520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57414749 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33238407 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5463413707776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5463233294336 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133926767 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76512018 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296188205779456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚3021.497459255 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57414751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287109780320880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76522161 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296188100718080 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290879446144000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.wÇlarge_pool.allocated‚290879463183360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296188100718080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19380987 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5308654574080 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76522161 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57414749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5308742596096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57414751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283307195664384ƽòÝ'ù×O TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33238404 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277843757052928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52619391 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296188100718080 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290879446144000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290879463183360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296188100718080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19380987 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5308654574080 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76522161 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57414749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5308742596096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57414751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283307195664384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚610 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292401833437525 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5292156304479 trainer/global_step‚7549 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5308654574080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19381237 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290879463183360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287109763281520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57414749 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33238407 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5463413707776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5463233294336 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133926767 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76512018 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296188205779456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚3021.497459255 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57414751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287109780320880 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76522161 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133954145 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5463922826752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19383731 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5292832256035 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5309420712960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133943986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33242697 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277879477113856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19383472 trainer/global_step‚7549 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283343588735488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292439381456729 % _timestamp‚1.7844999478340886e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76532073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290916829550592 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚611 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5309420712960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290916829550592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76532073 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296226145195008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296226250263552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133954145 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52626166 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 _runtime‚3021.745160793 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133943986 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290916812511232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57422072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277879452210176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57422072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290916812511232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808Q>Àç&üáM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5292832256035 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5309420712960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133943986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33242697 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277879477113856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19383472 trainer/global_step‚7549 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283343588735488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292439381456729 % _timestamp‚1.7844999478340886e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76532073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290916829550592 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚611 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5309420712960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290916829550592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76532073 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296226145195008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296226250263552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133954145 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52626166 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 _runtime‚3021.745160793 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133943986 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290916812511232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57422072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277879452210176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57422072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290916812511232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133954145 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5463922826752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19383731 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816ãõî8G2ý- _wandbruntime‚3021  _runtime‚3021‰Žˆ_þþÿ 0loss/FBt1/assignment_loss‚0.0053455508967116 /loss/FBt2/detection_loss‚0.6277950315923204 _runtime‚3021.745610834 /loss/FRt1/assignment_loss‚2.225267409774264 3loss/FBt2/assignment_loss‚0.0006499435254919593 .loss/FRt1/detection_loss‚0.583145250963128 % _timestamp‚1.7844999478350177e+09 2loss/SRqqt2/assignment_loss‚0.5128840065470877 0loss/SRqqt2/detection_loss‚0.568615319810233 /loss/FBt1/detection_loss‚0.6384089692257389 epoch‚9 trainer/global_step𪓂7549 /loss/FRt2/detection_loss‚1.4015859128866235 1loss/SRqqt1/detection_loss‚1.1113141636310173 %loss/total_loss‚9.324202137781098 _step‚612 0loss/FRt2/assignment_loss‚0.9629263078026912 2loss/SRqqt1/assignment_loss‚0.6862642711257912ßwF! ÿ„ /loss/FRt2/detection_loss‚1.4015859128866235 1loss/SRqqt1/detection_loss‚1.1113141636310173 %loss/total_loss‚9.324202137781098 _step‚612 0loss/FRt2/assignment_loss‚0.9629263078026912 2loss/SRqqt1/assignment_loss‚0.6862642711257912 0loss/FBt1/assignment_loss‚0.0053455508967116 /loss/FBt2/detection_loss‚0.6277950315923204 _runtime‚3021.745610834 /loss/FRt1/assignment_loss‚2.225267409774264 3loss/FBt2/assignment_loss‚0.0006499435254919593 .loss/FRt1/detection_loss‚0.583145250963128 % _timestamp‚1.7844999478350177e+09 2loss/SRqqt2/assignment_loss‚0.5128840065470877 0loss/SRqqt2/detection_loss‚0.568615319810233 /loss/FBt1/detection_loss‚0.6384089692257389 epoch‚9 trainer/global_step‚7549äw b€‚(:× ôõÒÐ’Ä+ network.recv‚897816) disk./.usagePercent‚76.23505423849753 disk.nvme0n1.in‚ 16.38671875 disk.nvme1n1.in‚3.25& disk.nvme1n1.out‚11085.98291015625 disk.dm-1.out‚2667.212890625 disk.rbd0.out‚0$ memory_percent‚99.21905994415283! disk.nvme0c0n1.in‚ 16.38671875& disk.nvme0c0n1.out‚19470.669921875 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚11085.98291015625 disk.dm-1.in‚0* proc.memory.percent‚17.501449584960938 cpu‚26.334072817860225 network.sent‚ 1.0034645e+07% disk./.usageGB‚1170.3986701965332 disk.dm-0.in‚ 19.63671875# disk.dm-0.out‚27697.05615234375 proc.memory.rssMB‚ 2867.4375 proc.cpu.threads‚17$ disk.nvme0n1.out‚19470.669921875 disk.md0.in‚0 disk.md0.out‚ 90.16015625 disk.rbd0.in‚0( proc.memory.availableMB‚ 127.94921875óff›‚(: ôõÒdž¥, gpu.0.gpu‚71.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.836' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.278666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú‚{Ò2‚- _wandbruntime‚3039  _runtime‚3039Ù7¢±ÆƒÀ & lr-AdamW/pg2‚9.440942965501756e-05 trainer/global_step‚7599 $ _timestamp‚1.784499966084359e+09 _runtime‚3039.995298761 _step‚613 & lr-AdamW/pg1‚9.440942965501756e-05©ÓæßË„Å _runtime‚3039.995298761 _step‚613 & lr-AdamW/pg1‚9.440942965501756e-05 & lr-AdamW/pg2‚9.440942965501756e-05 trainer/global_step‚7599 $ _timestamp‚1.784499966084359e+09å!CÈ2…- _wandbruntime‚3039  _runtime‚3039ЋØ'†ÒO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5342592315904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77008770 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3039.999355938 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298090461882880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292747781544960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279628539819008 _step‚614 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5342680337920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5498160200192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134789569 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288953383708880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844999660851865e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298090356821504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5342592315904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77008770 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3039.999355938 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298090461882880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292747781544960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279628539819008 _step‚614 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5342680337920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5498160200192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134789569 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288953383708880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7844999660851865e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298090356821504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57788122 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134806788 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _step‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844999663364172e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5498858924032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134806788 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134816947 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77018668 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57788120 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134816947 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52964793 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294316814320137 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288990255776640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292785147912192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285163270112256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77028825 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77028825 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57788120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5343270425600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298128401298432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279664234976256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5343270425600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52965047 LDDeviceStatsMonitor.on_train_batcht⑨ _end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19507599 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326575582857 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ûrÙç&ŠáM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77028825 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77028825 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57788120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5343270425600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298128401298432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279664234976256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5343270425600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52965047 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19507599 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326575582857 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285163068996608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57788122 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134806788 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _step‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844999663364172e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5498858924032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134806788 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134816947 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77018668 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57788120 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134816947 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52964793 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294316814320137 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288990255776640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292785147912192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285163270112256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ç…[2‹- _wandbruntime‚3040  _runtime‚3040Aü挂 0loss/FRt2/assignment_loss‚0.9689609892523608 /loss/FRt1/detection_loss‚0.5419777450470027 epoch‚9 2loss/SRqqt1/assignment_loss‚0.8860717994543031 1loss/SRqqt2/assignment_loss‚0.571879087068643 3loss/FBt2/assignment_loss‚0.0008812248402689095 /loss/FRt2/detection_loss‚1.4402314529368847 1loss/SRqqt1/detection_loss‚1.2291311156543834 &loss/total_loss‚10.482721636269785 trainer/global_step‚7599 % _timestamp‚1.7844999663373463e+09 0loss/FRt1/assignment_loss‚2.2957566762070276 3loss/FBt1/assignment_loss‚0.0006898983222873608 1loss/SRqqt2/detection_loss‚0.5393124188240034 /loss/FBt1/detection_loss‚1.4385794432212908 -loss/FBt2/detection_loss‚0.56924978544133 _runtime‚ 3040.24805247 _step‚616s©ç ‡ _runtime‚ 3040.24805247 _step‚616 0loss/FRt2/assignment_loss‚0.9689609892523608 /loss/FRt1/detection_loss‚0.5419777450470027 epoch‚9 2loss/SRqqt1/assignment_loss‚0.8860717994543031 1loss/SRqqt2/assignment_loss‚0.571879087068643 3loss/FBt2/assignment_loss‚0.0008812248402689095 /loss/FRt2/detection_loss‚1.4402314529368847 1loss/SRqqt1/detection_loss‚1.2291311156543834 &loss/total_loss‚10.482721636269785 trainer/global_step‚7599 % _timestamp‚1.7844999663373463e+09 0loss/FRt1/assignment_loss‚2.2957566762070276 3loss/FBt1/assignment_loss‚0.0006898983222873608 1loss/SRqqt2/detection_loss‚0.5393124188240034 /loss/FBt1/detection_loss‚1.4385794432212908 -loss/FBt2/detection_loss‚0.56924978544133èÐPqà\Ž‚(:Ñ ƒžõÒþ¬«+ disk.dm-1.out‚2695.404296875 disk.rbd0.out‚0 proc.memory.rssMB‚ 2867.4375 disk.nvme1c1n1.in‚3.25 disk.md0.out‚ 90.59765625 disk.dm-0.in‚ 19.63671875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.38671875 disk.nvme0n1.in‚ 16.38671875# disk.nvme0n1.out‚19478.03515625$ memory_percent‚99.21927452087402 network.recv‚902738' disk.nvme1c1n1.out‚11116.2080078125% disk.nvme0c0n1.out‚19478.03515625' proc.memory.availableMB‚ 127.9140625* proc.memory.percent‚17.501449584960938 cpu‚26.332355893903074 proc.cpu.threads‚17) disk./.usagePercent‚76.23494068587414% disk.nvme1n1.out‚11116.2080078125 disk.dm-1.in‚0 network.sent‚ 1.0096926e+07% disk./.usageGB‚1170.3969268798828 disk.nvme1n1.in‚3.25 disk.md0.in‚0" disk.dm-0.out‚27705.5126953125ÎrÌš‚(: ƒžõÒÉŸû+ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚195.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.30733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€¤b/2- _wandbruntime‚3059  _runtime‚3059/ˆÙŠÄ‘¾ $ _timestamp‚1.784499985153947e+09 _runtime‚3059.064871441 _step‚617 % lr-AdamW/pg1‚9.43036709121586e-05 % lr-AdamW/pg2‚9.43036709121586e-05 trainer/global_step‚7649Â\ªÉ’à % lr-AdamW/pg1‚9.43036709121586e-05 % lr-AdamW/pg2‚9.43036709121586e-05 trainer/global_step‚7649 $ _timestamp‚1.784499985153947e+09 _runtime‚3059.064871441 _step‚617éôÓûe2“- _wandbruntime‚3059  _runtime‚3059–ºqä×'”ÑO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77515665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33667407 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784499985154987e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77515665 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296156699164341 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5359815255295 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294616082867200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290796970057520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58146851 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5533288222208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299992612924928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19628789 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58146851 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286946585903616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281413272777728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299992717986304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚618 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286946370100736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294616099906560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77505522 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5359729106821 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53296456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33667404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77505522 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5376618079744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5376530057728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290796987096880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299992717986304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135652371 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5376618079744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58146849 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5533097323008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚4826g€°™!•ÖO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58146851 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5533288222208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299992612924928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19628789 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58146851 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286946585903616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281413272777728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299992717986304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚618 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286946370100736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294616099906560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77505522 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5359729106821 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53296456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33667404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77505522 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5376618079744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5376530057728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290796987096880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299992717986304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135652371 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5376618079744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58146849 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5533097323008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77515665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33667407 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784499985154987e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77515665 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonkC ˆCitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296156699164341 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5359815255295 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294616082867200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290796970057520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482êhJPº2–- _wandbruntime‚3059  _runtime‚3059ãyÈá&—ÛM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77515420 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58154172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135679749 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5533980209152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135679749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294653466273792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19631525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5533795608576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58154170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5377296196608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290833842125280 % _timestamp‚1.7844999854384017e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286982763543552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300030762470400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53303222 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286982973047808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294653449234432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281448992838656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53302963 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135669590 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296194247183545 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300030657401856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290833859164640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300030762470400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58154170 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77515420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33671697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚7649Ðmæ&˜àM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5377208167424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135669590 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5377208167424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77525577 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135669590 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296194247183545 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300030657401856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290833859164640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300030762470400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58154170 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77515420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33671697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚7649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77515420 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58154172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135679749 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5533980209152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135679749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294653466273792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19631525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5533795608576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58154170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5377296196608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290833842125280 % _timestamp‚1.7844999854384017e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286982763543552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300030762470400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53303222 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286982973047808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294653449234432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281448992838656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53302963 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292640573445520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5393644730703 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19753290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5410555821568 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53635197 % _timestamp‚1.7845000061342123e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296484418268160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296484401228800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78002274 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136525318 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298034235215583 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78012417 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78002274 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5393558582229 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19753037 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78012417 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDevicjùl¶eStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5410467799552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288766090928128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚301894974089728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33881904 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301894869028352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53634941 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58512899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301894974089728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58512901 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0»ðøÞÝ'¥×O `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292640590484880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301894869028352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53634941 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58512899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301894974089728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58512901 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292640573445520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5393644730703 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19753290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5410555821568 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53635197 % _timestamp‚1.7845000061342123e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296484418268160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296484401228800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78002274 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136525318 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298034235215583 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78012417 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78002274 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5393558582229 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19753037 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78012417 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5410467799552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288766090928128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚301894974089728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33881904 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136542551 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301933018573824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78012172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5411233938432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5411145909248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298071783234899 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301932913505280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78022329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33886194 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283233750701056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5568754260480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5568945152512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7699 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136532392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78012172 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78022329 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292677445513280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58520222 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5411145909248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53641634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288802480057856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58520220 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136542551 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296521767596032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53641883VªÐì6"¨áM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136542551 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301933018573824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78012172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5411233938432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5411145909248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298071783234899 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301932913505280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78022329 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33886194 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283233750701056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5568754260480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5568945152512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7699 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136532392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78012172 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78022329 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚623 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292677445513280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58520222 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5411145909248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53641634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288802480057856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58520220 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136542551 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296521767596032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53641883 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BäëǯÁ & lr-AdamW/pg1‚9.409006851504295e-05 & lr-AdamW/pg2‚9.409006851504295e-05 trainer/global_step‚7749 % _timestamp‚1.7845000261645906e+09 _runtime‚3100.075531193 _step‚625¦=áÌ°Æ % _timestamp‚1.7845000261645906e+09 _runtime‚3100.075531193 _step‚625 & lr-AdamW/pg1‚9.409006851504295e-05 & lr-AdamW/pg2‚9.409006851504295e-05 trainer/global_step‚7749ñïê-2±- _wandbruntime‚3100  _runtime‚3100KaØ'²ÒO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53973532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5444405541376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78499026 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58878949 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78499026 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78509169 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299911564891157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137377975 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137388120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303797230193152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5444493563392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34096407 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3100.076361725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19877370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19877128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303797230193152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34096404 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78509169 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284982788502528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚7749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298352719590400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303797125131776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298352736629760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427474206111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58878951 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7845000261655118e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303797125131776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5444493563392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299911668078991 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5427388057637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5427388057637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78499026 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58878949 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78499026 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78509169 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299911564891157 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137377975 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137388120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303797230193152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5444493563392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34096407 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3100.076361725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19877370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19877128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303797230193152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34096404 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78509169 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284982788502528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚7749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298352719590400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303797125131776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298352736629760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427474206111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58878951 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816òs„Aò2´- _wandbruntime‚3100  _runtime‚3100²C€;ÛµÛM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298390085957632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285018483659776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137405353 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53980567 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303835169608704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299949216098307 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428150157667 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58886272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5603761555968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78508924 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚7749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78508924 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303835274677248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5445171680256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294521048901280 _runtime‚3100.672821339 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137405353 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34100694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299949112910361 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298390102996992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78519081 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 $ _timestamp‚1.784500026761674e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137395194 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58886272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58886270 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5445083651072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285018508563456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298390085957632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303835274677248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53980294 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚627 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5445171680256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19879600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290622245215744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34100697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290622452622848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 56039440593925ŽŸæ&¶àM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137405353 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34100694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299949112910361 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298390102996992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78519081 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 $ _timestamp‚1.784500026761674e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137395194 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58886272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58886270 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5445083651072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285018508563456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298390085957632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303835274677248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53980294 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚627 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5445171680256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19879600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290622245215744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34100697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290622452622848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5603944059392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298390085957632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285018483659776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137405353 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53980567 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303835169608704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299949216098307 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428150157667 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58886272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5603761555968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78508924 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚7749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78508924 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303835274677248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5445171680256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294521048901280 _runtime‚3100.672821339óóظ2·- _wandbruntime‚3100  _runtime‚31004î5ܸ trainer/global_step‚7749 % _timestamp‚1.7845000267628074e+09 2loss/FBt1/assignment_loss‚0.015203019566782456 1loss/SRqqt2/detection_loss‚0.6718836442671781 /loss/FBt2/detection_loss‚0.4533385660732165 _step‚628 0loss/FRt1/assignment_loss‚2.1532408917306833 0loss/FRt2/assignment_loss‚1.0449764265792783 /loss/FRt1/detection_loss‚0.5785314527923601 /loss/FRt2/detection_loss‚1.5766537354567445 2loss/SRqqt1/assignment_loss‚0.6771223940375832 2loss/SRqqt2/assignment_loss‚0.6133853255043672 2loss/FBt2/assignment_loss‚0.004031027992987544 1loss/SRqqt1/detection_loss‚1.0272698798246922 /loss/FBt1/detection_loss‚0.9700340343963315 %loss/total_loss‚9.785670398222205 epoch‚9 _runtime‚ 3100.6734971(u ¹† 1loss/SRqqt2/detection_loss‚0.6718836442671781 /loss/FBt2/detection_loss‚0.4533385660732165 _step‚628 0loss/FRt1/assignment_loss‚2.1532408917306833 0loss/FRt2/assignment_loss‚1.0449764265792783 /loss/FRt1/detection_loss‚0.5785314527923601 /loss/FRt2/detection_loss‚1.5766537354567445 2loss/SRqqt1/assignment_loss‚0.6771223940375832 2loss/SRqqt2/assignment_loss‚0.6133853255043672 2loss/FBt2/assignment_loss‚0.004031027992987544 1loss/SRqqt1/detection_loss‚1.0272698798246922 /loss/FBt1/detection_loss‚0.9700340343963315 %loss/total_loss‚9.785670398222205 epoch‚9 _runtime‚ 3100.6734971 trainer/global_step‚7749 % _timestamp‚1.7845000267628074e+09 2loss/FBt1/assignment_loss‚0.015203019566782456ôÒXß^fº‚(:Û ¿žõÒœ§ö+ disk.nvme1c1n1.in‚3.25& disk.nvme1n1.out‚11123.36474609375 disk.md0.in‚0 disk.md0.out‚ 92.0703125 disk.rbd0.out‚0* proc.memory.percent‚17.501449584960938 proc.cpu.threads‚17 network.sent‚ 1.0300461e+07 network.recv‚919472) disk./.usagePercent‚76.05798842755149% disk./.usageGB‚1167.6802673339844( disk.nvme1c1n1.out‚11123.36474609375# disk.dm-0.out‚27721.07275390625 disk.dm-1.in‚0" disk.dm-1.out‚2699.43408203125 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 19.63671875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.38671875 disk.nvme0n1.in‚ 16.38671875& disk.nvme0n1.out‚19493.65185546875$ memory_percent‚99.22010898590088 proc.memory.rssMB‚ 2867.4375( disk.nvme0c0n1.out‚19493.65185546875( proc.memory.availableMB‚ 127.77734375 cpu‚26.33204359153241°ÛF,š»‚(: ¿žõÒÞ‹¹+ gpu.0.gpu‚54.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.538' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.51266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l+e¼‚(:Ú ΞõÒäÿ¬+& disk.nvme0n1.out‚19494.64111328125( proc.memory.availableMB‚ 127.76953125 proc.memory.rssMB‚ 2867.4375* proc.memory.percent‚17.501449584960938) disk./.usagePercent‚76.04251122983266% disk./.usageGB‚1167.4426536560059& disk.nvme1n1.out‚11124.47900390625 disk.md0.out‚ 92.6171875# disk.dm-0.out‚27721.53759765625 disk.rbd0.in‚0# memory_percent‚99.2201566696167 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚11124.47900390625 disk.dm-0.in‚ 19.63671875 proc.cpu.threads‚17 network.sent‚ 1.0350819e+07 disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.38671875( disk.nvme0c0n1.out‚19494.64111328125 disk.nvme0n1.in‚ 16.38671875 cpu‚26.32990543289529 network.recv‚923038 disk.dm-1.in‚0" disk.dm-1.out‚2699.90673828125œkÐY›½‚(: ΞõÒáâØ+ gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚183.077' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚61.025666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÒ•@2¾- _wandbruntime‚3120  _runtime‚3120ÇXˆ³Ç¿Á trainer/global_step‚7799 % _timestamp‚1.7845000464133012e+09 _runtime‚3120.324238473 _step‚629 & lr-AdamW/pg1‚9.398222828206221e-05 & lr-AdamW/pg2‚9.398222828206221e-05Ò4µ§ÌÀÆ % _timestamp‚1.7845000464133012e+09 _runtime‚3120.324238473 _step‚629 & lr-AdamW/pg1‚9.398222828206221e-05 & lr-AdamW/pg2‚9.398222828206221e-05 trainer/global_step‚7799õágņ2Á- _wandbruntime‚3120  _runtime‚3120hƒðØ'ÂÒO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286767571268608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚305699381235200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296327797260880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305699381235200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20000454 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300221037952000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5461217533045 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34310904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5478343283200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59244999 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301788997754565 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5478431305216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286767546364928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5638047084032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59245001 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59244999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79005921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300221054991360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5478343283200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78995778 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54311361 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5638229594624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305699486296576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34310907 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360/›«Ð—Ã×O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79005921 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5478431305216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300221037952000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292405593448960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 trainer/global_step‚7799 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5478431305216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286767546364928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5638047084032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59245001 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59244999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79005921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300221054991360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5478343283200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78995778 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54311361 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5638229594624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305699486296576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34310907 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286767571268608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚305699381235200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296327797260880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305699381235200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20000454 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300221037952000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5461217533045 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34310904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5478343283200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59244999 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301788997754565 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79005676 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5461979633075 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59252320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286803241522176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138268155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138257996 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34315194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300258404319232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305737530780672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚7799 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54317886 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34315197 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305737530780672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300258421358592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5479109422080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292442199250944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292441983455232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300258404319232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296364652289280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138268155 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300258421358592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5638741933056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5479109422080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5479021392896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845000466883574e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54318141 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59252322 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248¤ºç&ÆáM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59252322 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79005676 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5461979633075 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59252320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286803241522176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138268155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138257996 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34315194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300258404319232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305737530780672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚7799 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54317886 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34315197 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305737530780672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300258421358592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5479109422080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292442199250944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292441983455232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300258404319232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296364652289280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138268155 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300258421358592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5638741933056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5479109422080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5479021392896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845000466883574e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54318141 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0÷ ‰÷2Ç- _wandbruntime‚3120  _runtime‚31207"§†È€ .loss/FRt1/detection_loss‚0.632578068083155 trainer/global_step‚7799 % _timestamp‚1.7845000466893663e+09 2loss/FBt2/assignment_loss‚0.002926717349312283 /loss/FRt2/detection_loss‚1.5687941505393685 /loss/FBt1/detection_loss‚0.8777555253817065 .loss/FBt2/detection_loss‚0.594204256056295 _step‚632 0loss/FRt1/assignment_loss‚2.1976194733757595 0loss/FRt2/assignment_loss‚1.0450619098519076 1loss/SRqqt1/detection_loss‚1.0277934863534608 &loss/total_loss‚10.062549620758144 _runtime‚3120.600021812 epoch‚9 2loss/SRqqt1/assignment_loss‚0.7193274134859016 2loss/SRqqt2/assignment_loss‚0.6682949232616177 0loss/FBt1/assignment_loss‚0.0881610194312259 1loss/SRqqt2/detection_loss‚0.6400326775884341Tëö´ É… 2loss/FBt2/assignment_loss‚0.002926717349312283 /loss/FRt2/detection_loss‚1.5687941505393685 /loss/FBt1/detection_loss‚0.8777555253817065 .loss/FBt2/detection_loss‚0.594204256056295 _step‚632 0loss/FRt1/assignment_loss‚2.1976194733757595 0loss/FRt2/assignment_loss‚1.0450619098519076 1loss/SRqqt1/detection_loss‚1.0277934863534608 &loss/total_loss‚10.062549620758144 _runtime‚3120.600021812 epoch‚9 2loss/SRqqt1/assignment_loss‚0.7193274134859016 2loss/SRqqt2/assignment_loss‚0.6682949232616177 0loss/FBt1/assignment_loss‚0.0881610194312259 1loss/SRqqt2/detection_loss‚0.6400326775884341 .loss/FRt1/detection_loss‚0.632578068083155 trainer/global_step‚7799 % _timestamp‚1.7845000466893663e+09ø¨ê^\aÊ‚(:Ö ÝžõÒ¼Èè+ disk.nvme1c1n1.in‚3.25 disk.md0.in‚0 network.recv‚926515 disk.dm-1.in‚0 disk.nvme0n1.in‚ 16.38671875$ memory_percent‚99.22049045562744( proc.memory.availableMB‚ 127.71484375 proc.memory.rssMB‚ 2867.4375 proc.cpu.threads‚17 network.sent‚ 1.0366579e+07% disk./.usageGB‚1168.4469032287598" disk.dm-1.out‚2700.00830078125( disk.nvme1c1n1.out‚11125.51318359375& disk.nvme1n1.out‚11125.51318359375" disk.dm-0.out‚27726.0986328125* proc.memory.percent‚17.501449584960938 cpu‚26.327769458062964) disk./.usagePercent‚76.10792400122193 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.38671875 disk.nvme1n1.in‚3.25 disk.md0.out‚ 93.328125 disk.dm-0.in‚ 19.63671875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚19499.767578125$ disk.nvme0n1.out‚19499.767578125˜<ø™Ë‚(:Ž ÝžõÒ›¾‘, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.448' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.14933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#1îX2Ì- _wandbruntime‚3140  _runtime‚3140z¿Ô¼ÇÍÁ & lr-AdamW/pg1‚9.387369765881127e-05 & lr-AdamW/pg2‚9.387369765881127e-05 trainer/global_step‚7849 % _timestamp‚1.7845000666555202e+09 _runtime‚3140.566475563 _step‚633š|õÌÎÆ _runtime‚3140.566475563 _step‚633 & lr-AdamW/pg1‚9.387369765881127e-05 & lr-AdamW/pg2‚9.387369765881127e-05 trainer/global_step‚7849 % _timestamp‚1.7845000666555202e+09ùWFw2Ï- _wandbruntime‚3140  _runtime‚3140L†öE)ÐÒO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5512281025024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚307601742400000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307601637338624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34525407 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5512369047040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚7849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5512369047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303666533805807 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79502673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302089373352960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59611049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20124750 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5673234924544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302089373352960wû`¬Ý'Ñ×O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚7849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5512369047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303666533805807 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79502673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302089373352960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59611049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20124750 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5673234924544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302089373352960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5512281025024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚307601742400000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307601637338624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34525407 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5512369047040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303ú¨n„ô2Ò- _wandbruntime‚3140  _runtime‚3140• ¿Pâ&ÓÜM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5495722959897 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288587999384576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307639786884096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298208272716640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298208255677280 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139120798 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139130957 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302126739720192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288588024288256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294261963790848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302126722680832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5495809108483 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5512959134720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59618370 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 trainer/global_step‚7849 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139130957 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302126739720192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79502428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54656620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20126926 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845000669320357e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79502428 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307639681815552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294261750092288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59618372 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79512585 _runtime‚3140.843137049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302126722680832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5513047163904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303704081825123 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5512959134720 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚635 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59618372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20127180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360žu%ó=ÔáM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20127180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5495722959897 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288587999384576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307639786884096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298208272716640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298208255677280 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139120798 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139130957 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302126739720192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288588024288256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294261963790848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302126722680832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5495809108483 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5512959134720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59618370 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 trainer/global_step‚7849 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139130957 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonit s¿–ª or.on_train_batch_end/active_bytes.large_pool.allocated‚302126739720192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79502428 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54656620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20126926 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845000669320357e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79502428 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307639681815552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294261750092288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59618372 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79512585 _runtime‚3140.843137049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302126722680832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5513047163904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303704081825123 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5512959134720 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚635 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59618372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ûS¯îî2Õ- _wandbruntime‚3140  _runtime‚3140¡ó[Y Ö„ 0loss/FRt2/assignment_loss‚0.9607298614598122 3loss/FBt2/assignment_loss‚0.0015183620134015048 /loss/FRt2/detection_loss‚1.5393668078081761 0loss/FBt2/detection_loss‚0.45705829713648805 $ _timestamp‚1.784500066933135e+09 0loss/FRt1/assignment_loss‚2.3734624978267784 2loss/SRqqt1/assignment_loss‚0.7074906747488273 2loss/SRqqt2/assignment_loss‚0.6157458047499125 1loss/FBt1/assignment_loss‚0.04255282671914012 /loss/FRt1/detection_loss‚0.5483032339724072 1loss/SRqqt1/detection_loss‚1.2530257748676685 trainer/global_step‚7849 _runtime‚3140.843793251 _step‚636 &loss/total_loss‚10.100461565158103 epoch‚9 1loss/SRqqt2/detection_loss‚0.7847527593054461 /loss/FBt1/detection_loss‚0.8164546645500443XxŠ×‰ &loss/total_loss‚10.100461565158103 epoch‚9 1loss/SRqqt2/detection_loss‚0.7847527593054461 /loss/FBt1/detection_loss‚0.8164546645500443 0loss/FRt2/assignment_loss‚0.9607298614598122 3loss/FBt2/assignment_loss‚0.0015183620134015048 /loss/FRt2/detection_loss‚1.5393668078081761 0loss/FBt2/detection_loss‚0.45705829713648805 $ _timestamp‚1.784500066933135e+09 0loss/FRt1/assignment_loss‚2.3734624978267784 2loss/SRqqt1/assignment_loss‚0.7074906747488273 2loss/SRqqt2/assignment_loss‚0.6157458047499125 1loss/FBt1/assignment_loss‚0.04255282671914012 /loss/FRt1/detection_loss‚0.5483032339724072 1loss/SRqqt1/detection_loss‚1.2530257748676685 trainer/global_step‚7849 _runtime‚3140.843793251 _step‚636ü"0PŽaØ‚(:Ö ìžõÒص×+$ disk./.usageGB‚1170.203296661377 disk.nvme1n1.in‚3.25* proc.memory.percent‚17.501449584960938 cpu‚26.325822308342012 proc.cpu.threads‚17 network.sent‚ 1.0416058e+07 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21414.25732421875& disk.nvme0n1.out‚21414.25732421875& disk.nvme1n1.out‚11126.31689453125 disk.md0.out‚ 93.8515625 disk.dm-0.in‚ 19.63671875" disk.dm-0.out‚29640.1025390625 network.recv‚930846! disk.nvme0c0n1.in‚ 16.38671875" memory_percent‚99.220871925354( proc.memory.availableMB‚ 127.65234375 proc.memory.rssMB‚ 2867.4375) disk./.usagePercent‚76.22232839351095 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.38671875 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚11126.31689453125 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚2700.185546875Ë AšÙ‚(: ìžõÒµÂ, gpu.0.gpu‚54.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚149.065' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚49.68833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I`‘ƒ2Ú- _wandbruntime‚3160  _runtime‚3160TÇ ¥ÇÛÁ & lr-AdamW/pg1‚9.376447838362833e-05 & lr-AdamW/pg2‚9.376447838362833e-05 trainer/global_step‚7899 % _timestamp‚1.7845000869151967e+09 _runtime‚3160.826134713 _step‚637ÿ,ZÏÌÜÆ _runtime‚3160.826134713 _step‚637 & lr-AdamW/pg1‚9.376447838362833e-05 & lr-AdamW/pg2‚9.376447838362833e-05 trainer/global_step‚7899 % _timestamp‚1.7845000869151967e+09ý²¡û™2Ý- _wandbruntime‚3160  _runtime‚3160õòÃ…Ø'ÞÒO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59977099 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 trainer/global_step‚7899 % _timestamp‚1.7845000869162765e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79999425 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚303957691714560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546218766848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚303957674675200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34739904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚303957674675200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5546306788864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290337062089728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546218766848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5528962632335 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59977101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79999425 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296045294343680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296045084832256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305543966669215 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79989282 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20248856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20248608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5708207350272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79989282 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5708022742528 _runtime‚3160.827239225 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309503998503424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59977101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309503998503424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309503893442048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54988763 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139966381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚303957691714560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚638 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5546306788864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305543863481381 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34739907 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5528876483861 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546218766848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5528962632335 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59977101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79999425 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296045294343680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296045084832256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305543966669215 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79989282 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20248856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20248608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5708207350272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79989282 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5708022742528 _runtime‚3160.827239225 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309503998503424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59977101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309503998503424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309503893442048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54988763 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139966381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚303957691714560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚638 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5546306788864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305543863481381 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34739907 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5528876483861 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54995294 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139983600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5529638583891 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5546896876544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296081690236928 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5546984905728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784500087190294e+09 H:DeviceStatsMonitor.ë¦CLon_train_batch_end/active.all.allocated‚ 139993759 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59984422 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79999180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59984422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34744194 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139993759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34744197 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5708719291392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚3161.101382946 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54995536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5708908086272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309542042987520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚303995058081792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚303995041042432o!Ùæ&âàM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784500087190294e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139993759 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59984422 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79999180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59984422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34744194 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139993759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34744197 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5708719291392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚3161.101382946 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54995536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5708908086272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309542042987520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚303995058081792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚303995041042432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296081476538368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54995294 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139983600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5529638583891 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5546896876544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296081690236928 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5546984905728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104ÿRKÿÍ2ã- _wandbruntime‚3161  _runtime‚3161>êä€ % _timestamp‚1.7845000871912868e+09 _runtime‚3161.101977718 /loss/FRt1/assignment_loss‚2.073106179213283 epoch‚9 _step‚640 0loss/FBt1/assignment_loss‚0.1666175967960852 /loss/FBt1/detection_loss‚0.7528117977220703 2loss/SRqqt1/assignment_loss‚0.6826436035503666 2loss/SRqqt2/assignment_loss‚0.6791369550868777 2loss/FBt2/assignment_loss‚0.014455157040920312 /loss/FRt2/detection_loss‚1.4303045713003406 1loss/SRqqt1/detection_loss‚1.2071729470943582 1loss/SRqqt2/detection_loss‚0.6512546397478384 /loss/FBt2/detection_loss‚0.7733171598799526 0loss/FRt2/assignment_loss‚0.9038838411486426 /loss/FRt1/detection_loss‚0.5382650710201223 %loss/total_loss‚9.872969519600858 trainer/global_step‚7899û9 ¥ å… _step‚640 0loss/FBt1/assignment_loss‚0.1666175967960852 /loss/FBt1/detection_loss‚0.7528117977220703 2loss/SRqqt1/assignment_loss‚0.6826436035503666 2loss/SRqqt2/assignment_loss‚0.6791369550868777 2loss/FBt2/assignment_loss‚0.014455157040920312 /loss/FRt2/detection_loss‚1.4303045713003406 1loss/SRqqt1/detection_loss‚1.2071729470943582 1loss/SRqqt2/detection_loss‚0.6512546397478384 /loss/FBt2/detection_loss‚0.7733171598799526 0loss/FRt2/assignment_loss‚0.9038838411486426 /loss/FRt1/detection_loss‚0.5382650710201223 %loss/total_loss‚9.872969519600858 trainer/global_step‚7899 % _timestamp‚1.7845000871912868e+09 _runtime‚3161.101977718 /loss/FRt1/assignment_loss‚2.073106179213283 epoch‚9€rhùžXæ‚(:Í ûžõÒÇÖÒ+ disk.nvme1c1n1.in‚3.25 cpu‚26.323813155497756 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 16.38671875$ disk.nvme1c1n1.out‚ 11129.2265625" disk.nvme1n1.out‚ 11129.2265625 disk.dm-1.in‚0$ memory_percent‚99.22494888305664* proc.memory.percent‚17.501449584960938 disk.nvme1n1.in‚3.25 disk.md0.in‚0& proc.memory.availableMB‚ 126.984375' disk.nvme0c0n1.out‚21425.1865234375 disk.md0.out‚ 94.28515625 disk.dm-0.in‚ 19.63671875" disk.dm-0.out‚29652.9423828125 disk.dm-1.out‚2700.240234375 proc.memory.rssMB‚ 2867.4375 proc.cpu.threads‚17 network.sent‚ 1.0478372e+07 network.recv‚935319) disk./.usagePercent‚76.22235746497033$ disk./.usageGB‚1170.203742980957! disk.nvme0c0n1.in‚ 16.38671875% disk.nvme0n1.out‚21425.1865234375lLëéç‚(:„ ûžõÒŒàÿ+ gpu.0.gpu‚53.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚201.078' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Êåœ]è‚(:Ò ŠŸõÒµíÈ+' disk./.usagePercent‚76.219288065503 disk.nvme0n1.in‚ 16.38671875 proc.memory.rssMB‚ 2867.4375 network.sent‚ 1.0541925e+07" disk.dm-1.out‚2700.36083984375 disk.rbd0.in‚0& disk.nvme0n1.out‚21920.46923828125 disk.nvme1c1n1.in‚3.25# disk.nvme1n1.out‚11502.57421875 disk.md0.out‚ 94.48828125 disk.dm-0.in‚ 19.63671875 network.recv‚939533 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21920.46923828125 disk.nvme1n1.in‚3.25' proc.memory.availableMB‚ 126.8984375 cpu‚26.321745230401767 proc.cpu.threads‚17% disk./.usageGB‚1170.1566200256348! disk.nvme0c0n1.in‚ 16.38671875% disk.nvme1c1n1.out‚11502.57421875 disk.md0.in‚0! disk.dm-0.out‚30521.005859375$ memory_percent‚99.22547340393066* proc.memory.percent‚17.501449584960938hxÁÛšé‚(: ŠŸõÒ÷Ãý+ gpu.0.gpu‚52.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚180.661' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.22033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±#l¼2ê- _wandbruntime‚3180  _runtime‚3180ŽêÃ×Äë¾ $ _timestamp‚1.784500106851911e+09 _runtime‚3180.762846044 _step‚641 % lr-AdamW/pg1‚9.36545722058818e-05 % lr-AdamW/pg2‚9.36545722058818e-05 trainer/global_step‚7949kŒÏÉìà % lr-AdamW/pg1‚9.36545722058818e-05 % lr-AdamW/pg2‚9.36545722058818e-05 trainer/global_step‚7949 $ _timestamp‚1.784500106851911e+09 _runtime‚3180.762846044 _step‚641Jâ¦2í- _wandbruntime‚3180  _runtime‚3180m[Ø'îÒO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307421399532623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140829183 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34954404 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311406149545472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311406254606848 trainer/global_step‚7949 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60343149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55326485 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80486034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301858607424880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20372078 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292121819952128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34954407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292121844855808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60343149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305826010076160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305826010076160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305825993036800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305825993036800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5742977715712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60343151 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5562792107743 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80486034 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297865011373568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5743166517760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚301858590385520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5580244530688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297864797667840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚642 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5580156508672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845001068528843e+09 _runtime‚3180.763757945 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140839328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5562705959269 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307421296344789 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5580156508672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311406254606848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311406149545472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60343151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80496177 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Í'ŠÃ¶ï×O F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5580244530688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297864797667840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚642 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5580156508672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845001068528843e+09 _runtime‚3180.763757945 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140839328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5562705959269 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307421296344789 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5580156508672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311406254606848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311406149545472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60343151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80496177 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307421399532623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGD¯žº¬'eviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140829183 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34954404 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311406149545472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311406254606848 trainer/global_step‚7949 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60343149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55326485 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80486034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301858607424880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20372078 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292121819952128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34954407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292121844855808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60343149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305826010076160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305826010076160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305825993036800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305825993036800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5742977715712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60343151 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5562792107743 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80486034 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297865011373568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5743166517760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚301858590385520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104‚eÁ•R2ð- _wandbruntime‚3181  _runtime‚3181üoêˆâ&ñÜM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305863376443392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80506089 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140856561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20374300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292157540013056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5580834618368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80495932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5743863116800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5563468059299 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34958697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60350470 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301895479492640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34958694 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚305863359404032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5580834618368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305863359404032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5580922647552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚3181.357729414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20374545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297901403129856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140856561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55332994 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60350472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55333242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297901185236992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307458947551939 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60350472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5580922647552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5743670127616 _step‚643 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845001074466574e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80495932 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305863376443392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307458844363993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140846402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311444299090944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140846402 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464W3Ž\ç&òáM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301895462453280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307458844363993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140846402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311444299090944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140846402 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305863376443392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80506089 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140856561 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20374300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292157540013056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5580834618368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80495932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5743863116800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5563468059299 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34958697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60350470 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301895479492640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34958694 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚305863359404032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5580834618368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305863359404032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5580922647552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚3181.357729414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20374545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297901403129856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140856561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55332994 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60350472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55333242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297901185236992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307458947551939 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60350472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5580922647552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5743670127616 _step‚643 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845001074466574e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80495932 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305863376443392 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313308405648896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5777939942912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5614182272512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60709201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80992929 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20495239 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309298729208197 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80982786 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299684517757440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5778120356352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303702193773520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307694311398400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313308510710272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35168904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313308405648896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5777939942912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5614182272512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60709201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80992929 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20495239 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309298729208197 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80982786 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299684517757440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5778120356352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303702193773520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307694311398400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313308510710272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35168904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚303739082880640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80992684 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5614772360192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60716520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299721121097728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141719363 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141709204 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60716522 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55670614 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313346450125824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303739065841280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚7999 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚307731677765632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293942297875456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5778634427392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313346555194368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚307731694804992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20497420 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293942272971776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597297534707 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚7999 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚307731677765632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293942297875456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5778634427392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313346555194368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚307731694804992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20497420 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293942272971776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597297534707 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚303739082880640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_ú,׎end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80992684 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5614772360192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60716520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299721121097728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141719363 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141709204 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60716522 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55670614 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313346450125824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303739065841280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816‡<‹´K2- _wandbruntime‚3201  _runtime‚3201Ò"Mn‚€ /loss/FBt2/detection_loss‚0.7604317774777442 0loss/FRt1/assignment_loss‚2.2687765239869186 2loss/SRqqt1/assignment_loss‚0.6843309495193858 0loss/FBt2/assignment_loss‚0.0646550433984971 1loss/SRqqt2/detection_loss‚0.7432818496911968 /loss/FBt1/detection_loss‚0.6505494051923354 epoch‚9 $ _timestamp‚1.784500127386049e+09 _step‚648 2loss/FBt1/assignment_loss‚0.015089373478242152 1loss/SRqqt1/detection_loss‚1.2055304460831633 %loss/total_loss‚9.995387341154046 _runtime‚3201.300393512 0loss/FRt2/assignment_loss‚0.8893009221405409 2loss/SRqqt2/assignment_loss‚0.5874776307390366 /loss/FRt1/detection_loss‚0.6248027152640061 /loss/FRt2/detection_loss‚1.5011607041829786 trainer/global_step‚7999SSüà ƒ… 0loss/FRt1/assignment_loss‚2.2687765239869186 2loss/SRqqt1/assignment_loss‚0.6843309495193858 0loss/FBt2/assignment_loss‚0.0646550433984971 1loss/SRqqt2/detection_loss‚0.7432818496911968 /loss/FBt1/detection_loss‚0.6505494051923354 epoch‚9 $ _timestamp‚1.784500127386049e+09 _step‚648 2loss/FBt1/assignment_loss‚0.015089373478242152 1loss/SRqqt1/detection_loss‚1.2055304460831633 %loss/total_loss‚9.995387341154046 _runtime‚3201.300393512 0loss/FRt2/assignment_loss‚0.8893009221405409 2loss/SRqqt2/assignment_loss‚0.5874776307390366 /loss/FRt1/detection_loss‚0.6248027152640061 /loss/FRt2/detection_loss‚1.5011607041829786 trainer/global_step‚7999 /loss/FBt2/detection_loss‚0.7604317774777442ˆåÒŽ‡c„‚(:Ø ¨ŸõÒ”‰ñ+$ memory_percent‚99.23040866851807( proc.memory.availableMB‚ 126.08984375* proc.memory.percent‚17.501449584960938) disk./.usagePercent‚76.23479532857726& disk.nvme0n1.out‚22143.32958984375 disk.nvme1n1.in‚3.25! disk.dm-1.out‚2700.7158203125 proc.memory.rssMB‚ 2867.4375 cpu‚26.321847732488557 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.38671875( disk.nvme0c0n1.out‚22143.32958984375' disk.nvme1c1n1.out‚11535.9951171875 proc.cpu.threads‚17 network.sent‚ 1.0620823e+07 disk.rbd0.in‚0 disk.nvme1c1n1.in‚3.25 disk.md0.in‚0 disk.md0.out‚ 95.203125 disk.dm-0.in‚ 19.63671875 disk.dm-1.in‚0 network.recv‚947238% disk./.usageGB‚1170.3946952819824 disk.nvme0n1.in‚ 16.38671875% disk.nvme1n1.out‚11535.9951171875# disk.dm-0.out‚30775.38623046875{bk,˜…‚(: ¨ŸõÒ‡˜, gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚110.99' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.99666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08),‡2†- _wandbruntime‚3221  _runtime‚3221k®äpŒ=‡†{ & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20461933314800262 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04483756050467491 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4886440932750702 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8521050214767456 +particle/specificity‚0.9100409150123596 % _timestamp‚1.7845001475716116e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18449510633945465 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17009584605693817 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9830743074417114 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4195914566516876 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03606554865837097 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8523930907249451 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8321427702903748 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08649329096078873 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0432359054684639 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6360520124435425 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333730697632 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9827713966369629 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14749939739704132 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8523930907249451 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8562256693840027 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6398468613624573 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17570693790912628 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7720439434051514 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8329058885574341 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_2‚0.7644424438476562 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6054962277412415 jet/accuracy_2_of_5‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.707631528377533 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6912253499031067 ,"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09068047255277634 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5514194369316101 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5257861018180847 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40822628140449524 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.178874671459198 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6502181887626648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9959220886230469 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6867208480834961 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14426369965076447 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030309859663248062 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5839449167251587 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03592215105891228 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1577659249305725 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7206305265426636 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1581353098154068 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6388956904411316 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20493091642856598 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.807243824005127 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18962208926677704 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6283013224601746 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.456167995929718 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3586466908454895 particle/accuracy_1_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7407549619674683 +jet/accuracy_1_of_1‚0.31713613867759705 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1852085292339325 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9077708125114441 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19233620166778564 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.560444176197052 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.997012197971344 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.63124680519104 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚8009 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2648209035396576 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18865230679512024 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 +jet/accuracy_3_of_4‚0.09375353902578354 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6108472347259521 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 )loss/val_total_loss‚9.208676992503852 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _step‚649 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.68351811170578 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.377271443605423 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9923918843269348 particle/accuracy_1_of_1‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04406469315290451 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15626375377178192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9185011982917786 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6786471605300903 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6388956904411316 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006483063101768494 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09166666865348816 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.95416659116745 jet/accuracy_1_of_5‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10569589585065842 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04359596222639084 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9953113794326782 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8228069543838501 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876543283462524 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06551449745893478 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01545223593711853 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5091146230697632 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19340546429157257 )jet/accuracy_1_of_4‚0.969404935836792 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948727488517761 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987956285476685 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2354273796081543 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚9 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7703602910041809 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.376187801361084 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4885091185569763 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07190518081188202 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09166666865348816 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36890023946762085 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4966003894805908 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1582910567522049 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937213063240051 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7342886328697205 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03592215105891228 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8371580839157104 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16203196346759796 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8055380582809448 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9802547693252563 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4589579403400421 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17111128568649292 ,jet/accuracy_4_of_4‚0.014682539738714695 particle/accuracy_2_of_3‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9860186576843262 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6592088341712952 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.642510712146759 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7480719685554504 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5925385355949402 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25736430287361145 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19233620166778564 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014050889760255814 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20284846425056458 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7456866502761841 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 *jet/accuracy_1_of_3‚0.9144115447998047 *jet/accuracy_2_of_4‚0.8112987279891968 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8302502036094666 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7986841201782227 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4766978323459625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5398975014686584 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3012547791004181 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6164610385894775 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 .particle/accuracy_4_of_4‚0.959618866443634 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7294871211051941 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8972500562667847 *jet/accuracy_2_of_3‚0.5409970283508301 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *particle/sensitivity‚0.640239417552948 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.739088237285614 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969671368598938 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6158240437507629 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 jet/accuracy_5_of_5‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 _runtime‚3221.482986067 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9661484956741333 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7394595146179199 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938017725944519 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988871812820435 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6300846934318542 +jet/accuracy_2_of_2‚0.14141178131103516 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 /particle/accuracy_3_of_4‚0.9981684684753418 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9946196675300598 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982743263244629 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7017200589179993 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04495890066027641 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1óëc¬7$ˆ‹{ 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04406469315290451 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15626375377178192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9185011982917786 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6786471605300903 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6388956904411316 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006483063101768494 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09166666865348816 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.95416659116745 jet/accuracy_1_of_5‚1 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10569589585065842 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04359596222639084 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9953113794326782 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8228069543838501 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876543283462524 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06551449745893478 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01545223593711853 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5091146230697632 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19340546429157257 )jet/accuracy_1_of_4‚0.969404935836792 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948727488517761 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987956285476685 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2354273796081543 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚9 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7703602910041809 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.376187801361084 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4885091185569763 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07190518081188202 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09166666865348816 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.36890023946762085 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4966003894805908 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1582910567522049 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937213063240051 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7342886328697205 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03592215105891228 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8371580839157104 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16203196346759796 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8055380582809448 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9802547693252563 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4589579403400421 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17111128568649292 ,jet/accuracy_4_of_4‚0.014682539738714695 particle/accuracy_2_of_3‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9860186576843262 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6592088341712952 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.642510712146759 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7480719685554504 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5925385355949402 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25736430287361145 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19233620166778564 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014050889760255814 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20284846425056458 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7456866502761841 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 *jet/accuracy_1_of_3‚0.9144115447998047 *jet/accuracy_2_of_4‚0.8112987279891968 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8302502036094666 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7986841201782227 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4766978323459625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5398975014686584 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3012547791004181 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6164610385894775 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 .particle/accuracy_4_of_4‚0.959618866443634 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7294871211051941 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8972500562667847 *jet/accuracy_2_of_3‚0.5409970283508301 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *particle/sensitivity‚0.640239417552948 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.739088237285614 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969671368598938 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6158240437507629 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 jet/accuracy_5_of_5‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 _runtime‚3221.482986067 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9661484956741333 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7394595146179199 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938017725944519 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988871812820435 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6300846934318542 +jet/accuracy_2_of_2‚0.14141178131103516 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 /particle/accuracy_3_of_4‚0.9981684684753418 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9946196675300598 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982743263244629 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7017200589179993 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04495890066027641 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20461933314800262 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04483756050467491 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4886440932750702 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8521050214767456 +particle/specificity‚0.9100409150123596 % _timestamp‚1.7845001475716116e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18449510633945465 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17009584605693817 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9830743074417114 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4195914566516876 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03606554865837097 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8523930907249451 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8321427702903748 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08649329096078873 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0432359054684639 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6360520124435425 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333730697632ŽGeDZ 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9827713966369629 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14749939739704132 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8523930907249451 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8562256693840027 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6398468613624573 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17570693790912628 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7720439434051514 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8329058885574341 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_2‚0.7644424438476562 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6054962277412415 jet/accuracy_2_of_5‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.707631528377533 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6912253499031067 ,"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09068047255277634 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5514194369316101 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5257861018180847 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40822628140449524 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.178874671459198 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6502181887626648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9959220886230469 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6867208480834961 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14426369965076447 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030309859663248062 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5839449167251587 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03592215105891228 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1577659249305725 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7206305265426636 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1581353098154068 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6388956904411316 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20493091642856598 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.807243824005127 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18962208926677704 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6283013224601746 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.456167995929718 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3586466908454895 particle/accuracy_1_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7407549619674683 +jet/accuracy_1_of_1‚0.31713613867759705 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1852085292339325 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9077708125114441 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19233620166778564 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.560444176197052 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.997012197971344 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.63124680519104 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚8009 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2648209035396576 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18865230679512024 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 +jet/accuracy_3_of_4‚0.09375353902578354 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6108472347259521 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 )loss/val_total_loss‚9.208676992503852 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _step‚649 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.68351811170578 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.377271443605423 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9923918843269348 particle/accuracy_1_of_1‚1‰€7´Ùb‰‚(:× ·ŸõÒÊÞÈ+ network.sent‚ 1.0670214e+07 network.recv‚951668 disk.nvme1n1.in‚3.25 proc.memory.rssMB‚ 2870.90625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 16.38671875 disk.md0.in‚0 disk.dm-0.in‚ 19.63671875! disk.dm-1.out‚2764.6533203125 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.38671875( disk.nvme0c0n1.out‚22157.03759765625 cpu‚26.33101658986917# disk./.usageGB‚1167.67818069458' disk.nvme1c1n1.out‚11602.6513671875% disk.nvme1n1.out‚11602.6513671875 disk.md0.out‚ 95.60546875' proc.memory.availableMB‚ 192.2890625) disk./.usagePercent‚76.05785251226706& disk.nvme0n1.out‚22157.03759765625 disk.nvme1c1n1.in‚3.25# disk.dm-0.out‚30790.96142578125 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚98.82636070251465* proc.memory.percent‚17.522621154785156ŸÕÑŠ‚(:„ ·ŸõÒ¯ëñ+ gpu.0.gpu‚53.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚120.303' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05:Áþ2‹- _wandbruntime‚3238  _runtime‚32387îËÆŒÀ & lr-AdamW/pg1‚9.343270619515395e-05 & lr-AdamW/pg2‚9.343270619515395e-05 trainer/global_step‚8049 $ _timestamp‚1.784500164350041e+09 _runtime‚3238.260966322 _step‚650s€ËÅ trainer/global_step‚8049 $ _timestamp‚1.784500164350041e+09 _runtime‚3238.260966322 _step‚650 & lr-AdamW/pg1‚9.343270619515395e-05 & lr-AdamW/pg2‚9.343270619515395e-05Š©£j!2Ž- _wandbruntime‚3238  _runtime‚3238{ÕäÿØ'ÒO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61312719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35477484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20701201 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81646804 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316203850725888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚8049 _runtime‚3238.261848265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5833418334720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316203745664512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5668540331520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5668540331520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81656947 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61312717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142959521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302450390981120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312165045547523 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5668452309504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5668452309504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310535293355008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142969666 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56178685 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5650845066125 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310535310394368 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306514303669232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302450177275392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296616758940672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310535310394368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35477487 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5833607136768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81656947 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚306514286629872 _step‚651 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5650758917651 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56178943 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316203745664512 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1031878343¸.qÝ'×O `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306514303669232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302450177275392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296616758940672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310535310394368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35477487 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5833607136768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81656947 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚306514286629872 _step‚651 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5650758917651 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56178943 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316203745664512 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61312719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35477484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20701201 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81646804 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316203850725888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚8049 _runtime‚3238.261848265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5833418334720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316203745664512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5668540331520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5668540331520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81656947 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61312717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142959521 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302450390981120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312165045547523 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5668452309504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5668452309504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310535293355008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142969666 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56178685 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5650845066125 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310535310394368‹1ÓX2‘- _wandbruntime‚3238  _runtime‚3238ä5è+Œ’ÛM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142986899 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56185503 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20703729 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310572676761600 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310572659722240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5669130419200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35481774 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5669218448384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142976740 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5651521017681 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81656702 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306551175736992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5834120797184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306551158697632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316241895209984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5651434869095 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310572676761600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20703995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316241895209984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61320040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296652479001600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312202696754673 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5669218448384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35481777 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296652454097920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚8049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5669130419200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61320038 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303÷§jÀæ&“àM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81666859 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316241790141440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302486574895104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310572676761600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20703995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316241895209984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61320040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296652479001600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312202696754673 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5669218448384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35481777 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296652454097920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚8049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5669130419200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61320038 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142986899 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56185503 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20703729 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310572676761600 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310572659722240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5669130419200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35481774 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5669218448384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142976740 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5651521017681 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81656702 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306551175736992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5834120797184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306551158697632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316241895209984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5651434869095 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61678769 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298401541706752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143822323 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61678767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35691987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318106001767936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5702478073344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61678769 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82153699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚655 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312403611716608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312403628755968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20826449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308357890017872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318106106829312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20826719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312403628755968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312403611716608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5868406477824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5684674541533 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318106001767936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298401516803072 # _timestamp‚1.78450018438817e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82153699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712œû÷Ìb ÔO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304270136986624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5684588393059 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5702478073344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314042581598765 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314042478410931 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61678769 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298401541706752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143822323 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61678767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35691987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318106001767936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5702478073344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61678769 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82153699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚655 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312403611716608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312403628755968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20826449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308357890017872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318106106829312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20826719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312403628755968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312403611716608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5868406477824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5684674541533 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318106001767936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298401516803072 # _timestamp‚1.78450018438817e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82153699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82143556 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318106106829312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273·ŽÐ2¡- _wandbruntime‚3258  _runtime‚3258ƒ®©á&¢ÛM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20829146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298437236864000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312440995123200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5703156190208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35696274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304306524720128 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318144151313408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61686090 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61686088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143839542 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143849701 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5869111644160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5685264344503 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82163611 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚656 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308394779124992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3258.574411952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5703156190208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61686088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5685350493089 $ _timestamp‚1.784500184663335e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314080026430135 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314080129618081 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318144151313408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312440995123200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56525167 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143839542ˆÍYæ&£àM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35696274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304306524720128 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318144151313408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61686090 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61686088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143839542 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143849701 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5869111644160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5685264344503 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82163611 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚656 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308394779124992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3258.574411952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5703156190208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61686088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5685350493089 $ _timestamp‚1.784500184663335e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314080026430135 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314080129618081 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318144151313408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312440995123200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56525167 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143839542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20829146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298437236864000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312440995123200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5703156190208+Š%2¤- _wandbruntime‚3258  _runtime‚3258~µzÊ¥‚ /loss/FRt1/detection_loss‚0.5246687959950247 /loss/FRt2/detection_loss‚1.3878199251522119 epoch‚10 trainer/global_step‚8099 _step‚657 0loss/FRt2/assignment_loss‚1.0033598816796634 2loss/SRqqt1/assignment_loss‚0.7064532728559948 0loss/SRqqt1/detection_loss‚1.020122849887473 1loss/SRqqt2/detection_loss‚0.7015212617647657 /loss/FBt1/detection_loss‚0.8314830626582261 _runtime‚3258.575041903 0loss/FRt1/assignment_loss‚2.2090549961927066 $ _timestamp‚1.784500184664395e+09 2loss/FBt1/assignment_loss‚0.024207356530922368 /loss/FBt2/detection_loss‚0.8045923441767375 %loss/total_loss‚9.887328950841972 2loss/SRqqt2/assignment_loss‚0.6715498942608616 2loss/FBt2/assignment_loss‚0.002495309687384043Y‡1 ¦‡ 2loss/FBt1/assignment_loss‚0.024207356530922368 /loss/FBt2/detection_loss‚0.8045923441767375 %loss/total_loss‚9.887328950841972 2loss/SRqqt2/assignment_loss‚0.6715498942608616 2loss/FBt2/assignment_loss‚0.002495309687384043 /loss/FRt1/detection_loss‚0.5246687959950247 /loss/FRt2/detection_loss‚1.3878199251522119 epoch‚10 trainer/global_step‚8099 _step‚657 0loss/FRt2/assignment_loss‚1.0033598816796634 2loss/SRqqt1/assignment_loss‚0.7064532728559948 0loss/SRqqt1/detection_loss‚1.020122849887473 1loss/SRqqt2/detection_loss‚0.7015212617647657 /loss/FBt1/detection_loss‚0.8314830626582261 _runtime‚3258.575041903 0loss/FRt1/assignment_loss‚2.2090549961927066 $ _timestamp‚1.784500184664395e+09‘¹œ7¶`§‚(:Õ äŸõÒí´Ù+' disk.nvme1c1n1.out‚13442.6005859375' disk.nvme0c0n1.out‚23951.6533203125 proc.cpu.threads‚17% disk.nvme0n1.out‚23951.6533203125 disk.nvme1n1.in‚3.25 disk.md0.in‚0! disk.dm-1.out‚2828.8818359375 disk.rbd0.out‚0* proc.memory.percent‚17.522621154785156% disk./.usageGB‚1171.1606483459473 disk.nvme1c1n1.in‚3.25% disk.nvme1n1.out‚13442.6005859375 disk.md0.out‚ 96.8828125 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375$ memory_percent‚98.82473945617676 network.sent‚ 1.0794263e+07 network.recv‚963795 disk.nvme0n1.in‚ 16.43359375 disk.dm-0.in‚ 19.68359375" disk.dm-0.out‚34358.5517578125' proc.memory.availableMB‚ 192.5546875 proc.memory.rssMB‚ 2870.90625 cpu‚26.325106633914807( disk./.usagePercent‚76.2846864253996@ÍÒ™¨‚(:Ž äŸõÒ÷éµ, gpu.0.gpu‚55.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.21' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.07000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V›§2©- _wandbruntime‚3278  _runtime‚3278ÿ¿÷ǪÁ _runtime‚3278.301284815 _step‚658 & lr-AdamW/pg1‚9.320811384110876e-05 & lr-AdamW/pg2‚9.320811384110876e-05 trainer/global_step‚8149 % _timestamp‚1.7845002043904226e+09Ç““Ì«Æ _runtime‚3278.301284815 _step‚658 & lr-AdamW/pg1‚9.320811384110876e-05 & lr-AdamW/pg2‚9.320811384110876e-05 trainer/global_step‚8149 % _timestamp‚1.7845002043904226e+09’Ê„0x2¬- _wandbruntime‚3278  _runtime‚3278©?ö­ÒO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 trainer/global_step‚8149 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320008257871360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35906484 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5718504016941 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82640308 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82640308 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5718417868467 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚659 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320008257871360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5736327793152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5736327793152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20951165 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315919911274339 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56857649 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314271947117568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5903600189952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144685125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320008362932736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320008362932736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5736415815168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5736415815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82650451 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62044819 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300186274665472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314271947117568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5903415582208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚3278.302088787 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144695270 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310201493405872 % _timestamp‚1.7845002043912904e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82650451 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314271930078208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306089899759104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35906487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20951419 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315920014462173 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310201510445232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62044819 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314271930078208 A7Ý'®×O TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35906484 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5718504016941 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82640308 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82640308 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5718417868467 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚659 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320008257871360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5736327793152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5736327793152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20951165 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315919911274339 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56857649 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314271947117568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5903600189952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144685125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320008362932736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320008362932736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5736415815168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5736415815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82650451 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62044819 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300186274665472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314271947117568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5903415582208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚3278.302088787 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144695270 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310201493405872 % _timestamp‚1.7845002043912904e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82650451 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314271930078208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306089899759104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35906487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20951419 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315920014462173 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310201510445232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62044819 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314271930078208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 trainer/global_step‚8149 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320008257871360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0“¾ó©22¯- _wandbruntime‚3278  _runtime‚3278›Áâ&°ÜM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35910777 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300221994726400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306126085550080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5904296133632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚8149 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62052138 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 _runtime‚3278.561736031 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144712503 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56864424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314309296445440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5737093932032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300221969822720 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144702344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20953650 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144712503 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62052140 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20953911 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚660 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82660363 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314309313484800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56864688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5904115727360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320046407416832 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314309313484800 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310238382512992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144702344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82660363 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35910774 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62052138 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5719179968497 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104°”– ±áM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5737005902848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5719093819911 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5719179968497 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35910777 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300221994726400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306126085550080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5904296133632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚8149 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62052138 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 _runtime‚3278.561736031 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144712503 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56864424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314309296445440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5737093932032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300221969822720 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144702344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20953650 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144712503 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62052140 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20953911 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚660 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82660363 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314309313484800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56864688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5904115727360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320046407416832 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314309313484800 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310238382512992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144702344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82660363 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35910774 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62052138 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚312045096793872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5752333492349 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 trainer/global_step‚8199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321910513974784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62410867 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83137060 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770265534976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5938432061952 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83137060 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57195526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62410869 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317797344137747 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145547927 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317797447325581 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5752247343875 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316140265479168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770265534976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36120987 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚301971032527872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5770353556992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316140248439808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62410869 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36120984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21074542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316140265479168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321910513974784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21074790 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784500223462492e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62410867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5938610378240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317797344137747 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145547927 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317797447325581 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5752247343875 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316140265479168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770265534976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36120987 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚301971032527872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5770353556992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316140248439808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62410869 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36120984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21074542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316140265479168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321910513974784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21074790 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784500223462492e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62410867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5938610378240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚312045096793872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5752333492349 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 trainer/global_step‚8199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321910513974784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62410867 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83137060 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770265534976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5938432061952 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83137060 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57195526 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374—*;»Î2½- _wandbruntime‚3297  _runtime‚3297‘©h&ª¾ÜM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145575305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316177631846400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21077266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5939133902848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_traiÁúݤ8n_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317834892156951 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62418188 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83157115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57202290 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145565146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321948558451712 _step‚664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62418188 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5939310114816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5771031673856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302006727685120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145575305 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5770943644672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57202543 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321948663520256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316177614807040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312081968861632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5771031673856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83146958 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36125277 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845002237157893e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62418190 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317834995344897 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5752923295319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321948663520256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5770943644672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83146958 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307945861587968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145565146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62418190 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482ZæÆfç&¿áM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5753009443905 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚8199 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317834892156951 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62418188 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83157115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57202290 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145565146 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321948558451712 _step‚664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62418188 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5939310114816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5771031673856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302006727685120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145575305 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5770943644672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57202543 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321948663520256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316177614807040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312081968861632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5771031673856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83146958 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36125277 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845002237157893e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62418190 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317834995344897 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5752923295319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321948663520256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5770943644672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83146958 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307945861587968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145565146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62418190 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145575305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316177631846400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21077266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5939133902848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B2Ç- _wandbruntime‚3316  _runtime‚3316óJÆÈÀ _runtime‚3316.274615059 _step‚666 & lr-AdamW/pg1‚9.298080953289827e-05 & lr-AdamW/pg2‚9.298080953289827e-05 trainer/global_step‚8249 $ _timestamp‚1.784500242363692e+09ëÃ,ËÉÅ & lr-AdamW/pg1‚9.298080953289827e-05 & lr-AdamW/pg2‚9.298080953289827e-05 trainer/global_step‚8249 $ _timestamp‚1.784500242363692e+09 _runtime‚3316.274615059 _step‚666šye­@2Ê- _wandbruntime‚3316  _runtime‚3316ýUó[Ø'ËÒO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21199569 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319674777001155 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚3316.275534351 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318008566801408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5804291298816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5804203276800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309729376294400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62776919 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323812770078208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309729162588672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5804291298816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83633812 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62776917 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62776917 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83643955 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21199835 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313888700181872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303755815293952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318008583840768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323812875139584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318008566801408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146410729 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323812770078208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62776919 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5804203276800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚667 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323812770078208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62776919 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5804203276800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚667 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323812770078208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309729162588672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5804291298816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83633812 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62776917 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62776917 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83643955 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21199835 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313888700181872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303755815293952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318008583840768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323812875139584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318008566801408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808›‚¤ÇZ2Í- _wandbruntime‚3316  _runtime‚3316Ù¥0ðâ&ÎÜM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5804969415680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313925589288992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845002426232402e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83653867 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5804881386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303791485547520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146427948 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57541854 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62784238 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62784240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5786752770727 trainer/global_step‚8249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5804881386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318045950208000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309765766393856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5786838919313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62784238 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146438107 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21202080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5974255942656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323850814555136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323850919623680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5804969415680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5804969415680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313925589288992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845002426232402e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83653867 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5804881386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303791485547520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146427948 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57541854 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62784238 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62784240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5786752770727 trainer/global_step‚8249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5804881386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318045950208000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309765766393856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5786838919313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62784238 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146438107 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21202080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5974255942656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323850814555136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323850919623680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5804969415680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325715026181632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325715131243008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21324131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21323868 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6008315287552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321552209864563 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319876902202368 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147273531 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311548863540224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319876902202368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319876885163008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36549987 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305540573156352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311549079343104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5819906294691 _runtime‚3335.124440837 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5838141018624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845002612130196e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84140707 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36549984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84130564 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319876885163008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5838229040640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5838141018624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63142967 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84130564 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63142969 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147283676CéŒÝ'Ú×O VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36549984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84130564 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚319876885163008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5838229040640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5838141018624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63142967 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84130564 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63142969 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147283676 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5819992443165 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325715026181632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚8299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84140707 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325715026181632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325715131243008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21324131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21323868 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6008315287552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321552209864563 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319876902202368 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147273531 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311548863540224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319876902202368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319876885163008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36549987 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305540573156352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311549079343104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5819906294691 _runtime‚3335.124440837 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5838141018624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845002612130196e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84140707 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360Ÿ˜é·2Û- _wandbruntime‚3335  _runtime‚3335\o1Õâ&ÜÜM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84150619 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325753070658560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84140462 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311585465340416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63150290 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84140462 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319914251530240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63150288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315769192676992 trainer/global_step‚8299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147290750 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311585264224768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36554274 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63150288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6009020814848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21326357 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315769175637632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5820582246135 _step‚672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305576243409920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5820668394721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325753175727104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84150619 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5838907157504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36554277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57880890 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147300909 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147300909 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57880631 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321589861071713 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325753175727104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5838907157504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321589757883767 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319914268569600 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0s:«2ÝáM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84140462 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319914251530240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63150288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315769192676992 trainer/global_step‚8299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147290750 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311585264224768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36554274 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63150288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ;4DeviceStatsMoni–WÒ tor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6009020814848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21326357 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315769175637632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5820582246135 _step‚672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305576243409920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5820668394721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325753175727104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84150619 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5838907157504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36554277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57880890 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147300909 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147300909 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57880631 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321589861071713 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325753175727104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5838907157504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321589757883767 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319914268569600 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84150619 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325753070658560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84140462 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311585465340416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63150290 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 p³h2Þ- _wandbruntime‚3335  _runtime‚3335"< ÿßù 2loss/SRqqt1/assignment_loss‚0.6435100523598212 1loss/SRqqt2/assignment_loss‚0.593933633378548 .loss/FBt2/detection_loss‚0.690931637197112 %loss/total_loss‚9.704262532015104 % _timestamp‚1.7845002614719708e+09 _step‚673 0loss/FBt1/assignment_loss‚0.1246061451074036 /loss/FBt1/detection_loss‚0.7109957517761933 /loss/FRt2/assignment_loss‚1.094956451614431 0loss/SRqqt1/detection_loss‚1.102716572865961 epoch‚10 /loss/FRt1/assignment_loss‚2.252622396818229 2loss/FBt2/assignment_loss‚0.005438798427180321 .loss/FRt1/detection_loss‚0.553019851361114 ,loss/FRt2/detection_loss‚1.3178537434042 1loss/SRqqt2/detection_loss‚0.6136774977049095 trainer/global_step‚8299 _runtime‚3335.382609601œfUàþ /loss/FRt2/assignment_loss‚1.094956451614431 0loss/SRqqt1/detection_loss‚1.102716572865961 epoch‚10 /loss/FRt1/assignment_loss‚2.252622396818229 2loss/FBt2/assignment_loss‚0.005438798427180321 .loss/FRt1/detection_loss‚0.553019851361114 ,loss/FRt2/detection_loss‚1.3178537434042 1loss/SRqqt2/detection_loss‚0.6136774977049095 trainer/global_step‚8299 _runtime‚3335.382609601 2loss/SRqqt1/assignment_loss‚0.6435100523598212 1loss/SRqqt2/assignment_loss‚0.593933633378548 .loss/FBt2/detection_loss‚0.690931637197112 %loss/total_loss‚9.704262532015104 % _timestamp‚1.7845002614719708e+09 _step‚673 0loss/FBt1/assignment_loss‚0.1246061451074036 /loss/FBt1/detection_loss‚0.7109957517761933¡í+x‘gá‚(:Ü ¯ õÒ¦£›+ disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375& disk.nvme0n1.out‚30197.16943359375 disk.md0.in‚0 network.recv‚985221& disk.nvme1n1.out‚13579.34423828125 disk.md0.out‚ 99.05859375 disk.dm-0.in‚ 19.68359375# disk.dm-0.out‚40735.24853515625' proc.memory.availableMB‚ 192.4921875 proc.memory.rssMB‚ 2870.90625* proc.memory.percent‚17.522621154785156( disk.nvme0c0n1.out‚30197.16943359375 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚13579.34423828125 disk.nvme1n1.in‚3.25 disk.dm-1.in‚0 network.sent‚ 1.104854e+07" disk.dm-1.out‚2829.76708984375$ memory_percent‚98.82512092590332 cpu‚26.318821502996663 proc.cpu.threads‚17) disk./.usagePercent‚76.46161532716513% disk./.usageGB‚1173.8769493103027¶ºûù™â‚(:Ž ¯ õÒç–È+ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.617' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.20566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à|i©2ã- _wandbruntime‚3353  _runtime‚3353u'»-ÇäÁ & lr-AdamW/pg1‚9.275080783342363e-05 & lr-AdamW/pg2‚9.275080783342363e-05 trainer/global_step‚8349 % _timestamp‚1.7845002797352154e+09 _runtime‚3353.646128913 _step‚674uݦÌåÆ _runtime‚3353.646128913 _step‚674 & lr-AdamW/pg1‚9.275080783342363e-05 & lr-AdamW/pg2‚9.275080783342363e-05 trainer/global_step‚8349 % _timestamp‚1.7845002797352154e+09¢|åÂv2æ- _wandbruntime‚3353  _runtime‚3353½¼âØ'çÒO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5872166782464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84627316 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5853735770099 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36764484 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307325306115072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323429642727971 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84637459 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8349 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148136333 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327617282285056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845002797360914e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63509017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317575906957872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5853821918573 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84637459 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148146478 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313368801837056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚3353.646899444 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63509017 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323429745915805 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321745220563968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5872078760448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58212188 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327617282285056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36764487 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327617387346432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307325331018752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321745203524608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6043292502016 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5872078760448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58212443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21447704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313368598617088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5872166782464 _step‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327617387346432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321745203524608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360DbѬÝ'è×O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5872166782464 _step‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327617387346432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321745203524608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5872166782464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84627316 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5853735770099 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36764484 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307325306115072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323429642727971 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84637459 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8349 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148136333 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327617282285056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845002797360914e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63509017 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317575906957872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5853821918573 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84637459 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148146478 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313368801837056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚3353.646899444 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63509017 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323429745915805 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321745220563968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5872078760448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58212188 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327617282285056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36764487 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327617387346432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307325331018752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321745203524608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6043292502016 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5872078760448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58212443 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21447704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313368598617088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330£™IOÍ2é- _wandbruntime‚3353  _runtime‚3353yªÄF¹êÛM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63516338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58218873 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321782586931200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307361026176000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321782569891840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317612796064992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36768777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148153552 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327655431830528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 _step‚676 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6043984123392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84647371 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5872844899328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327655431830528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5854411721543 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313405203288576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612779025632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148163711 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323467293935121 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845002799916759e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148163711 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63516338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58218873 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321782586931200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307361026176000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321782569891840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317612796064992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36768777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148153552 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327655431830528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 _step‚676 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6043984123392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84647371 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5872844899328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327655431830528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5854411721543 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313405203288576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612779025632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148163711 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323467293935121 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845002799916759e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148163711 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248¤Ðä2ì- _wandbruntime‚3353  _runtime‚3353Á7Æíÿ /loss/FBt2/detection_loss‚0.4567799359720084 1loss/SRqqt1/assignment_loss‚0.771636735823482 1loss/SRqqt1/detection_loss‚1.2453040457833155 %loss/total_loss‚9.914422352949764 trainer/global_step‚8349 % _timestamp‚1.7845002799933898e+09 _step‚677 /loss/FRt1/assignment_loss‚2.188537346020436 0loss/FRt2/assignment_loss‚0.9949760235341868 2loss/SRqqt2/assignment_loss‚0.5870355040341875 epoch‚10 _runtime‚3353.904107582 1loss/FBt1/assignment_loss‚0.05209824056363805 0loss/FBt2/assignment_loss‚0.0041571598448229 /loss/FRt1/detection_loss‚0.5397083422306138 /loss/FBt1/detection_loss‚0.8784986381702449 /loss/FRt2/detection_loss‚1.4740709238983272 1loss/SRqqt2/detection_loss‚0.7216194570745024µýS î„ _step‚677 /loss/FRt1/assignment_loss‚2.188537346020436 0loss/FRt2/assignment_loss‚0.9949760235341868 2loss/SRqqt2/assignment_loss‚0.5870355040341875 epoch‚10 _runtime‚3353.904107582 1loss/FBt1/assignment_loss‚0.05209824056363805 0loss/FBt2/assignment_loss‚0.0041571598448229 /loss/FRt1/detection_loss‚0.5397083422306138 /loss/FBt1/detection_loss‚0.8784986381702449 /loss/FRt2/detection_loss‚1.4740709238983272 1loss/SRqqt2/detection_loss‚0.7216194570745024 /loss/FBt2/detection_loss‚0.4567799359720084 1loss/SRqqt1/assignment_loss‚0.771636735823482 1loss/SRqqt1/detection_loss‚1.2453040457833155 %loss/total_loss‚9.914422352949764 trainer/global_step‚8349 % _timestamp‚1.7845002799933898e+09¥vBàZï‚(:Ï ¾ õÒÄãÙ+ network.recv‚989224 disk.dm-1.in‚0 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25$ memory_percent‚98.82686138153076) disk./.usagePercent‚76.46138722802233$ disk./.usageGB‚1173.873447418213# disk.dm-0.out‚40745.42431640625 disk.nvme0n1.in‚ 16.43359375& disk.nvme0n1.out‚30203.69189453125 disk.nvme1n1.out‚ 13587.78125 disk.md0.out‚ 99.32421875( proc.memory.availableMB‚ 192.20703125 network.sent‚ 1.1124628e+07 disk.dm-0.in‚ 19.68359375 disk.rbd0.in‚0 proc.memory.rssMB‚ 2870.90625 cpu‚26.317099251879796 proc.cpu.threads‚17 disk.dm-1.out‚2833.966796875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375( disk.nvme0c0n1.out‚30203.69189453125" disk.nvme1c1n1.out‚ 13587.78125 disk.md0.in‚0* proc.memory.percent‚17.522621154785156öý?šð‚(: ¾ õÒÝ»…, gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.557' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.85233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ…02ñ- _wandbruntime‚3372  _runtime‚3372£7xÇòÁ & lr-AdamW/pg1‚9.263480006014898e-05 & lr-AdamW/pg2‚9.263480006014898e-05 trainer/global_step‚8399 % _timestamp‚1.7845002986224015e+09 _runtime‚3372.533312552 _step‚678õ0ÌóÆ & lr-AdamW/pg2‚9.263480006014898e-05 trainer/global_step‚8399 % _timestamp‚1.7845002986224015e+09 _runtime‚3372.533312552 _step‚678 & lr-AdamW/pg1‚9.263480006014898e-05¦I .ï2ô- _wandbruntime‚3372  _runtime‚3372+¨r&õÒO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149009280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85134211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5887651393981 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329519643449856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5906016502272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315188523024384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323613521886208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325307075591379 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚323613521886208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚323613538925568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5887565245507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63875069 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329519643449856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5906104524288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚8399 % _timestamp‚1.7845002986232417e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36978987 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6078253729792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5906016502272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329519538388480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21571102 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148999135 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329519538388480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63875067 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315188317707264 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309110088881152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319419527385232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3372.536327027 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63875069 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85124068 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58550345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323613538925568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85124068 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36978984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚325307178779213 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/ * Ûfsegment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5906104524288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482ó|±Ý'ö×O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36978987 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6078253729792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5906016502272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329519538388480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21571102 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148999135 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329519538388480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63875067 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315188317707264 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309110088881152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319419527385232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3372.536327027 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63875069 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85124068 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58550345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323613538925568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85124068 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36978984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚325307178779213 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5906104524288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149009280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85134211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5887651393981 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329519643449856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5906016502272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315188523024384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323613521886208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325307075591379 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚323613521886208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚323613538925568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5887565245507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63875069 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329519643449856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5906104524288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚8399 % _timestamp‚1.7845002986232417e+09§=~·¥2÷- _wandbruntime‚3372  _runtime‚3372ÇJ~€à&øÚM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58556800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329557687933952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149026513 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6078945330688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315224922358272 _step‚680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21573526 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329557582865408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63882388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21573784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323650888253440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85133966 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309145759134720 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85133966 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323650905292800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5906782641152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323650888253440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3372.790792873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329557687933952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5906694611968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325344726798529 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325344623610583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36983277 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63882390 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149016354 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5906694611968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36983274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5888241196951 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149016354 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6079138319872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63882388 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319456382413632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5888327345537 trainer/global_step‚8399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85144123 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315224704465408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6078945330688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315224922358272 _step‚680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21573526 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329557582865408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63882388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21573784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323650888253440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85133966 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309145759134720 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85133966 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323650905292800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5906782641152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323650888253440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3372.790792873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329557687933952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5906694611968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325344726798529 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325344623610583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36983277 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63882390 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149016354 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5906694611968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36983274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5888241196951 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149016354 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6079138319872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63882388 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319456382413632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5888327345537 trainer/global_step‚8399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85144123 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315224704465408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58556800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329557687933952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149026513 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331421794491904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321263113733872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85620820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327184508454787 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37193487 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58888186 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5921394720915 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331421899553280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149872082 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64241117 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚3391.455201788 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85630963 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331421794491904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 % _timestamp‚1.7845003175443833e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85620820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21694449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325481840247808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5939954244096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327184611642621 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64241119 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325481857287168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5921480869389 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64241119 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321263130773232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317008238305792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5940042266112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325481840247808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5939954244096 trainer/global_step‚8449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85630963 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5940042266112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310894846743552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310894821839872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325481840247808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5939954244096 trainer/global_step‚8449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85630963 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5940042266112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310894846743552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310894821839872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331421794491904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321263113733872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85620820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327184508454787 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37193487 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58888186 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5921394720915 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331421899553280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149872082 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64241117 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚3391.455201788 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85630963 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331421794491904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 % _timestamp‚1.7845003175443833e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85620820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21694449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325481840247808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5939954244096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327184611642621 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64241119 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325481857287168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5921480869389 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330«7¼m%2‡- _wandbruntime‚3391  _runtime‚3391³õ!&ˆÚM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325519206615040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5940632353792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85640875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37197777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21697112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58894652 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331459838968832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5940632353792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327222159661937 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5922070672359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331459944037376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5940720382976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3391.712859554 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6113909024768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327222056473991 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 # _timestamp‚1.78450031780169e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64248440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85640875 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331459944037376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317044631331840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310930516997120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6114089431040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321300002840992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5922156820945 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21696878 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149889315 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚684 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85630718 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.a"@l ¿llocated‚325519223654400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310930541900800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482d$¦å&‰ßM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚684 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85630718 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325519223654400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310930541900800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325519206615040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5940632353792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85640875 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37197777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21697112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58894652 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331459838968832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5940632353792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327222159661937 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5922070672359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331459944037376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5940720382976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3391.712859554 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6113909024768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327222056473991 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 # _timestamp‚1.78450031780169e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64248440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85640875 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331459944037376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317044631331840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310930516997120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6114089431040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321300002840992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5922156820945 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21696878 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149889315¬µH![2Š- _wandbruntime‚3391  _runtime‚33917q˜V‹ÿ 0loss/FBt1/assignment_loss‚0.0866656658403798 0loss/FBt2/assignment_loss‚0.0453469552526323 /loss/FRt1/detection_loss‚0.5929780757710278 /loss/FRt2/detection_loss‚1.4379822180018593 /loss/FBt1/detection_loss‚0.7188059524633063 epoch‚10 2loss/SRqqt1/assignment_loss‚0.7150087239018543 1loss/SRqqt1/detection_loss‚1.0651670722290874 1loss/SRqqt2/detection_loss‚0.6471199617093649 %loss/total_loss‚9.656344242887469 trainer/global_step‚8449 _runtime‚3391.715269977 2loss/SRqqt2/assignment_loss‚0.5808687135565208 $ _timestamp‚1.784500317803452e+09 /loss/FRt1/assignment_loss‚2.295090713112031 0loss/FRt2/assignment_loss‚0.9732436619330722 0loss/FBt2/detection_loss‚0.49806652911633376 _step‚685¤¼¹n Œ„ trainer/global_step‚8449 _runtime‚3391.715269977 2loss/SRqqt2/assignment_loss‚0.5808687135565208 $ _timestamp‚1.784500317803452e+09 /loss/FRt1/assignment_loss‚2.295090713112031 0loss/FRt2/assignment_loss‚0.9732436619330722 0loss/FBt2/detection_loss‚0.49806652911633376 _step‚685 0loss/FBt1/assignment_loss‚0.0866656658403798 0loss/FBt2/assignment_loss‚0.0453469552526323 /loss/FRt1/detection_loss‚0.5929780757710278 /loss/FRt2/detection_loss‚1.4379822180018593 /loss/FBt1/detection_loss‚0.7188059524633063 epoch‚10 2loss/SRqqt1/assignment_loss‚0.7150087239018543 1loss/SRqqt1/detection_loss‚1.0651670722290874 1loss/SRqqt2/detection_loss‚0.6471199617093649 %loss/total_loss‚9.656344242887469­c ždk‚(:à ë õÒð®‹,! disk.dm-1.out‚2834.4150390625 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚31995.43701171875 proc.memory.rssMB‚ 2870.90625 proc.cpu.threads‚17 network.sent‚ 1.1252602e+07 network.recv‚ 1.000903e+06 disk.nvme1c1n1.in‚3.25 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0 cpu‚26.312552600528846( disk.nvme1c1n1.out‚15362.97021484375 disk.md0.in‚0 disk.md0.out‚ 100.5390625" disk.dm-0.out‚44309.3037109375& disk.nvme0n1.out‚31995.43701171875) disk./.usagePercent‚76.46190430244081% disk./.usageGB‚1173.8813858032227 disk.nvme1n1.in‚3.25! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375$ memory_percent‚98.82745742797852& proc.memory.availableMB‚ 192.109375* proc.memory.percent‚17.522621154785156& disk.nvme1n1.out‚15362.97021484375xϲ̜Ž‚(:‘ ë õÒª¢£+ gpu.0.gpu‚85.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.239' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.413000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’z7 2- _wandbruntime‚3411  _runtime‚3411ÉfžÆÀ & lr-AdamW/pg1‚9.240077995699514e-05 & lr-AdamW/pg2‚9.240077995699514e-05 trainer/global_step‚8499 $ _timestamp‚1.784500337340872e+09 _runtime‚3411.251809325 _step‚686?=WêË‘Å & lr-AdamW/pg1‚9.240077995699514e-05 & lr-AdamW/pg2‚9.240077995699514e-05 trainer/global_step‚8499 $ _timestamp‚1.784500337340872e+09 _runtime‚3411.251809325 _step‚686®mRÄï2’- _wandbruntime‚3411  _runtime‚3411•Ø'“ÒO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333324050595328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64607167 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86127715 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21818618 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327350158609408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86117572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312679604605952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329061941318195 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚8499 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64607167 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845003373418357e+09 _runtime‚3411.252695266 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64607169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86127715 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329062044506029 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323106734161232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5973891985920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37407984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312679579702272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6148151653888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64607169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5955310344797 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚318827731356160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5973891985920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59226605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21818365 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327350158609408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0tTLì"”×O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333324050595328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64607167 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86127715 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21818618 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327350158609408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86117572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312679604605952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329061941318195 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚8499 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64607167 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845003373418357e+09 _runtime‚3411.252695266 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64607169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86127715 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329062044506029 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323106734161232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5973891985920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37407984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312679579702272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6148151653888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64607169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5955310344797 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚318827731356160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5973891985920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59226605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21818365 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327350158609408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816¯–“®õ2•- _wandbruntime‚3411  _runtime‚3411ªÐµMâ&–ÜM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333362200140800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5974570095616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327387542016000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312715299763200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329099489337399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5974658124800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845003376142914e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59233141 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚323143606228992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955986296353 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333362200140800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86127470 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6149034863616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327387524976640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318864125122560 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚3411.525451951 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327387542016000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150752117 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312715274859520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150741958 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329099592525345 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333362095072256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚8499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5955900147767 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86137627 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5974570095616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64614490 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21821111 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6148850263040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323143589189632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59233388 '#·ç&—áM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955986296353 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333362200140800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86127470 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6149034863616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327387524976640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318864125122560 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚3411.525451951 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327387542016000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150752117 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312715274859520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150741958 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329099592525345 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333362095072256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚8499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5955900147767 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86137627 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5974570095616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64614490 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21821111 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6148850263040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323143589189632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59233388 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333362200140800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5974570095616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327387542016000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312715299763200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329099489337399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5974658124800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845003376142914e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59233141 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚323143606228992°gâ‹2˜- _wandbruntime‚3411  _runtime‚3411Ÿ!‡™ÿ 2loss/SRqqt2/assignment_loss‚0.5354003866938332 /loss/FRt2/detection_loss‚1.3827705835801294 /loss/FBt1/detection_loss‚0.5724928638627452 trainer/global_step‚8499 _step‚689 /loss/FRt1/assignment_loss‚2.085050675687911 % _timestamp‚1.7845003376153595e+09 _runtime‚3411.526156802 1loss/SRqqt1/assignment_loss‚0.642665211301805 0loss/FBt1/assignment_loss‚0.1530931872779283 1loss/FBt2/assignment_loss‚0.00872685192554869 /loss/FRt1/detection_loss‚0.5707156433634714 %loss/total_loss‚9.408848070219621 epoch‚10 1loss/SRqqt1/detection_loss‚1.1604338288377745 1loss/SRqqt2/detection_loss‚0.6004792129381464 /loss/FBt2/detection_loss‚0.7070926412594535 0loss/FRt2/assignment_loss‚0.9899269834908739ON š„ _step‚689 /loss/FRt1/assignment_loss‚2.085050675687911 % _timestamp‚1.7845003376153595e+09 _runtime‚3411.526156802 1loss/SRqqt1/assignment_loss‚0.642665211301805 0loss/FBt1/assignment_loss‚0.1530931872779283 1loss/FBt2/assignment_loss‚0.00872685192554869 /loss/FRt1/detection_loss‚0.5707156433634714 %loss/total_loss‚9.408848070219621 epoch‚10 1loss/SRqqt1/detection_loss‚1.1604338288377745 1loss/SRqqt2/detection_loss‚0.6004792129381464 /loss/FBt2/detection_loss‚0.7070926412594535 0loss/FRt2/assignment_loss‚0.9899269834908739 2loss/SRqqt2/assignment_loss‚0.5354003866938332 /loss/FRt2/detection_loss‚1.3827705835801294 /loss/FBt1/detection_loss‚0.5724928638627452 trainer/global_step‚8499±ƒBV¶››‚(: ú õÒ¸Žµ+ gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.616' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.205333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åm²Ðfœ‚(:Û ú õÒá«ä+( proc.memory.availableMB‚ 192.07421875 proc.memory.rssMB‚ 2870.90625 network.sent‚ 1.1301481e+07 network.recv‚ 1.005211e+06 disk.nvme1n1.in‚3.25 disk.md0.in‚0# memory_percent‚98.8276720046997* proc.memory.percent‚17.522621154785156) disk./.usagePercent‚76.46164464709852# disk./.usageGB‚1173.87739944458" disk.dm-1.out‚2834.53271484375 disk.rbd0.out‚0 cpu‚26.312793956410562% disk.nvme0c0n1.out‚32000.66796875# disk.nvme0n1.out‚32000.66796875 disk.nvme1c1n1.in‚3.25 disk.md0.out‚ 101.07421875" disk.dm-0.out‚44314.5537109375 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375( disk.nvme1c1n1.out‚15364.24853515625& disk.nvme1n1.out‚15364.24853515625 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0w»‚2- _wandbruntime‚3431  _runtime‚3431K=:²Åž¿ trainer/global_step‚8549 % _timestamp‚1.7845003572721844e+09 _runtime‚3431.183123262 _step‚690 % lr-AdamW/pg1‚9.22827713754229e-05 % lr-AdamW/pg2‚9.22827713754229e-05tMÎ×ÊŸÄ % lr-AdamW/pg1‚9.22827713754229e-05 % lr-AdamW/pg2‚9.22827713754229e-05 trainer/global_step‚8549 % _timestamp‚1.7845003572721844e+09 _runtime‚3431.183123262 _step‚690²3 FŽ2 - _wandbruntime‚3431  _runtime‚3431ûÓÚ}¡ÒO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37622487 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335226411760128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚691 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335226306698752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329218476971008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6007829727744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151587541 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324950337549232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314464337564672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330939374181603 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6007917749760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324950320509872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5989053671731 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5989139820205 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86624467 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64973219 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329218476971008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6007917749760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314464362468352 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37622484 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21943044 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21942796 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86614324 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320647447066624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6183109501952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59565280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86614324 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86624467 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329218494010368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64973217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64973217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329218494010368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6007829727744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6183287818240 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚Xà»[482 trainer/global_step‚8549 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335226306698752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330939477369437 % _timestamp‚1.7845003572731879e+09 _runtime‚3431.184085984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335226306698752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329218476971008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6007829727744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151587541 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324950337549232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314464337564672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330939374181603 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6007917749760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324950320509872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5989053671731 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5989139820205 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86624467 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64973219 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329218476971008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6007917749760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314464362468352 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37622484 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21943044 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21942796 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86614324 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320647447066624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6183109501952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59565280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86614324 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86624467 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329218494010368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64973217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64973217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329218494010368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6007829727744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6183287818240 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚8549 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335226306698752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330939477369437 % _timestamp‚1.7845003572731879e+09 _runtime‚3431.184085984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5989729623175 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6008507837440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6008507837440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151614919 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86634379 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86624222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 trainer/global_step‚8549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330976922200807 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64980540 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335264456244224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335264456244224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64980540 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64980538 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21945312 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151604760 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21945564 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚320683840005632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151614919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329255843338240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59572086 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320684045315584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324987192577632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64980538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330977025388753 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324987209616992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314500057625600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335264351175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151604760k¨Î`ç&¥áM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330977025388753 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324987209616992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314500057625600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335264351175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151604760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37626774 _step‚692 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314500032721920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5989729623175 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6008507837440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6008507837440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151614919 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86634379 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86624222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 trainer/global_step‚8549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330976922200807 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64980540 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335264456244224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335264456244224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64980540 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64980538 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21945312 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151604760 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21945564 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚320683840005632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151614919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329255843338240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59572086 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320684045315584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324987192577632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64980538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303´Ûåt2¦- _wandbruntime‚3431  _runtime‚3431õEFj§ /loss/FRt2/detection_loss‚1.4223632044193242 %loss/total_loss‚9.598428697655127 trainer/global_step‚8549 _runtime‚3431.456639873 2loss/SRqqt2/assignment_loss‚0.6030799444447589 epoch‚10 0loss/FRt1/assignment_loss‚2.2303031159771813 0loss/FRt2/assignment_loss‚1.0089104960765345 1loss/FBt1/assignment_loss‚0.034075250šŒH)73751663 /loss/FRt1/detection_loss‚0.6359017153019765 1loss/SRqqt1/detection_loss‚1.1412475513856088 /loss/FBt2/detection_loss‚0.4325416608498647 _step‚693 1loss/SRqqt1/assignment_loss‚0.601129869249509 1loss/FBt2/assignment_loss‚0.00666729543141243 1loss/SRqqt2/detection_loss‚0.6934730769314158 /loss/FBt1/detection_loss‚0.7887355168500254 % _timestamp‚1.7845003575459669e+09É43 ¨† 2loss/SRqqt2/assignment_loss‚0.6030799444447589 epoch‚10 0loss/FRt1/assignment_loss‚2.2303031159771813 0loss/FRt2/assignment_loss‚1.0089104960765345 1loss/FBt1/assignment_loss‚0.03407525073751663 /loss/FRt1/detection_loss‚0.6359017153019765 1loss/SRqqt1/detection_loss‚1.1412475513856088 /loss/FBt2/detection_loss‚0.4325416608498647 _step‚693 1loss/SRqqt1/assignment_loss‚0.601129869249509 1loss/FBt2/assignment_loss‚0.00666729543141243 1loss/SRqqt2/detection_loss‚0.6934730769314158 /loss/FBt1/detection_loss‚0.7887355168500254 % _timestamp‚1.7845003575459669e+09 /loss/FRt2/detection_loss‚1.4223632044193242 %loss/total_loss‚9.598428697655127 trainer/global_step‚8549 _runtime‚3431.456639873µ¾V§g©‚(:Ü ‰¡õÒŸ¼Ö+ disk.nvme0n1.in‚ 16.43359375( disk.nvme1c1n1.out‚15365.89794921875 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 19.68359375& disk.nvme0c0n1.out‚32009.650390625( proc.memory.availableMB‚ 192.25390625 proc.cpu.threads‚17 network.sent‚ 1.1363825e+07% disk./.usageGB‚1173.8744354248047 disk.nvme1c1n1.in‚3.25 disk.dm-1.in‚0" disk.dm-1.out‚2834.58740234375 proc.memory.rssMB‚ 2870.90625 disk.md0.out‚ 101.421875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375* proc.memory.percent‚17.522621154785156 cpu‚26.311144950801747 network.recv‚ 1.009961e+06) disk./.usagePercent‚76.46145158279138$ disk.nvme0n1.out‚32009.650390625& disk.nvme1n1.out‚15365.89794921875 disk.md0.in‚0 disk.dm-0.out‚44324.39453125 disk.rbd0.out‚0$ memory_percent‚98.826575279235841š]Q›ª‚(: ‰¡õÒåò«, gpu.0.gpu‚60.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚181.686' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚60.562000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ųҼš«‚(: ˜¡õÒò‡Ä+ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚165.332' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚55.11066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂY[d¬‚(:Ù ˜¡õÒÜ‚Ÿ,& disk.nvme0n1.out‚32013.55517578125* proc.memory.percent‚17.522621154785156 cpu‚26.309210876187702 network.sent‚ 1.1426125e+07$ disk./.usageGB‚1172.137565612793 disk.dm-0.in‚ 19.68359375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚32013.55517578125% disk.nvme1c1n1.out‚15369.23046875 disk.nvme1n1.in‚3.25 disk.md0.out‚ 101.578125$ memory_percent‚98.82662296295166( proc.memory.availableMB‚ 192.24609375 proc.memory.rssMB‚ 2870.90625) disk./.usagePercent‚76.34831888049457# disk.nvme1n1.out‚15369.23046875 proc.cpu.threads‚17 network.recv‚ 1.014435e+06# disk.dm-0.out‚44331.13330078125 disk.dm-1.out‚ 2834.73046875 disk.rbd0.out‚0ñ]mP2­- _wandbruntime‚3451  _runtime‚3451™¡{^Ç®Á _runtime‚3451.390710203 _step‚694 & lr-AdamW/pg1‚9.216409962383225e-05 & lr-AdamW/pg2‚9.216409962383225e-05 trainer/global_step‚8599 % _timestamp‚1.7845003774797826e+09Z’,ZÌ¯Æ % _timestamp‚1.7845003774797826e+09 _runtime‚3451.390710203 _step‚694 & lr-AdamW/pg1‚9.216409962383225e-05 & lr-AdamW/pg2‚9.216409962383225e-05 trainer/global_step‚8599¶už¶2°- _wandbruntime‚3451  _runtime‚3451÷u¿Ø'±ÒO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87121219 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331086812371968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326793923897872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59904240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37836984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316249120330752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331086795332608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65339269 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337128667863552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331086812371968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6218098467328 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6218287269376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65339269 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 % _timestamp‚1.7845003774807425e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322467407600128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22067256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337128562802176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6041855491584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332816910232845 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3451.391599395 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59904506 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22067519 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326793940937232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331086795332608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65339267 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6041855491584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37836987 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316249095427072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6022883147139 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87111076 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337128562802176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6022969295613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65339267 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337128667863552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22067256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337128562802176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6041855491584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332816910232845 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3451.391599395 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59904506 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22067519 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326793940937232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331086795332608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65339267 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6041855491584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37836987 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316249095427072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6022883147139 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87111076 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337128562802176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6022969295613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65339267 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337128667863552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104·zü2³- _wandbruntime‚3451  _runtime‚3451ÉuµÅ´ÛM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6023559098583 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65346590 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 trainer/global_step‚8599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152467562 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65346590 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152477721 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6042445579264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87120974 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322503586681856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6042533608448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152477721 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87131131 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37841277 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22070017 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚ 3451.66347395 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6042445579264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337166607279104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332854458252161 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59911033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332854355064215 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87120974 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337166712347648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152467562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6023645247169 _step‚696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331124161699840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59911294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22069759 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326830813004992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6218980698112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326830795965632 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331124178739200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331124178739200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65346588 OADeviceStatsMonitor.o/#÷îÊn_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65346588 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87131131 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316284815488000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337166712347648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104‰#,ßæ&µàM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87131131 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316284815488000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337166712347648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6023559098583 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65346590 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 trainer/global_step‚8599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152467562 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65346590 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152477721 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6042445579264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87120974 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322503586681856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6042533608448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152477721 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87131131 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37841277 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22070017 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚ 3451.66347395 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6042445579264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337166607279104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332854458252161 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59911033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332854355064215 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87120974 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337166712347648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152467562 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6023645247169 _step‚696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331124161699840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59911294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22069759 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326830813004992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6218980698112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326830795965632 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331124178739200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331124178739200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65346588 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65346588 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339030818905600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60243238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328637544325232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65705317 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339030923966976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6075793233408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6075705211392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38051484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6075793233408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65705319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38051487 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153323290 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6056798771021 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332955113694208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334694343096253 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845003977794318e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87607828 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332955113694208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332955130733568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87617971 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6056712622547 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚8649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339030923966976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87617971 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332955130733568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6253253999104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22191754 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318033853289472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339030818905600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87607828 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334694239908419 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22192018´\åÀ×O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845003977794318e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87607828 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332955113694208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332955130733568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87617971 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6056712622547 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚8649 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339030923966976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87617971 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332955130733568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6253253999104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22191754 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318033853289472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339030818905600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87607828 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334694239908419 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22192018 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328637527285872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65705319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6253071488512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚3471.690340843 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60243505 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _step‚699 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318033878193152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339030818905600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60243238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328637544325232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMon£ÌFø itor.on_train_batch_start/active.large_pool.freed‚65705317 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339030923966976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6075793233408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6075705211392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38051484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6075793233408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65705319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38051487 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153323290 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6056798771021 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332955113694208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334694343096253 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0» é’2Á- _wandbruntime‚3471  _runtime‚3471ºŽL?á&ÂÛM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87627883 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87617726 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22194472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6057474722577 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332992497100800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38055774 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153340523 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318069573350400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65712640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60250249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65712638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚328674416392992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6057388573991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153340523 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚328674399353632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87617726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 3471.96368457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚8649 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87627883 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6076471350272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22194215 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324323312846848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332992497100800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6253955292160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6253764400128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚700 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339068968451072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845003980525923e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65712640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334731787927623 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60249989 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38055777 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332992480061440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324323528642560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153330364 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152·ˆ•æ&ÃàM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332992497100800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6253955292160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6253764400128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚700 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339068968451072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845003980525923e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65712640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334731787927623 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60249989 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38055777 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332992480061440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324323528642560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153330364 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87627883 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87617726 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22194472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6057474722577 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332992497100800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38055774 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153340523 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318069573350400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65712640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60250249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65712638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚328674416392992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6057388573991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153340523 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚328674399353632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87617726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 3471.96368457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚8649 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87627883 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6076471350272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22194215 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324323312846848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2¼¹BM2Ä- _wandbruntime‚3471  _runtime‚3471Ð¡í“ Å„ 3loss/FBt2/assignment_loss‚0.0015967525486225982 /loss/FRt2/detection_loss‚1.5516351567274784 /loss/FBt1/detection_loss‚0.6440894685313105 % _timestamp‚1.7845003980536742e+09 _step‚701 /loss/FRt2/assignment_loss‚0.998441958226232 2loss/SRqqt2/assignment_loss‚0.6079376078226204 /loss/FRt1/detection_loss‚0.5325307985287308 1loss/SRqqt1/detection_loss‚1.0480108220116608 /loss/FBt2/detection_loss‚0.6010235673440445 trainer/global_step‚8649 _runtime‚3471.964348902 epoch‚10 3loss/FBt1/assignment_loss‚0.0013129764258815158 1loss/SRqqt2/detection_loss‚0.6220305633270893 %loss/total_loss‚9.567716795014395 /loss/FRt1/assignment_loss‚2.180862556097536 2loss/SRqqt1/assignment_loss‚0.7782445674231888ñ¨üžÆ‰ /loss/FRt1/assignment_loss‚2.180862556097536 2loss/SRqqt1/assignment_loss‚0.7782445674231888 3loss/FBt2/assignment_loss‚0.0015967525486225982 /loss/FRt2/detection_loss‚1.5516351567274784 /loss/FBt1/detection_loss‚0.6440894685313105 % _timestamp‚1.7845003980536742e+09 _step‚701 /loss/FRt2/assignment_loss‚0.998441958226232 2loss/SRqqt2/assignment_loss‚0.6079376078226204 /loss/FRt1/detection_loss‚0.5325307985287308 1loss/SRqqt1/detection_loss‚1.0480108220116608 /loss/FBt2/detection_loss‚0.6010235673440445 trainer/global_step‚8649 _runtime‚3471.964348902 epoch‚10 3loss/FBt1/assignment_loss‚0.0013129764258815158 1loss/SRqqt2/detection_loss‚0.6220305633270893 %loss/total_loss‚9.567716795014395½oìC]Ç‚(:Ò ¶¡õÒ¡ÇÅ+ proc.cpu.threads‚17 network.recv‚ 1.022613e+06 disk.dm-1.in‚0 proc.memory.rssMB‚ 2870.90625# disk.dm-0.out‚47889.52978515625! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375! disk.nvme1c1n1.out‚ 15373.0625 disk.nvme1n1.in‚3.25 disk.nvme1n1.out‚ 15373.0625 disk.md0.out‚ 102.578125 cpu‚26.30570732969961( disk.nvme0c0n1.out‚35570.45361328125 network.sent‚ 1.1507146e+07% disk./.usageGB‚1173.8818740844727 disk.dm-0.in‚ 19.68359375 disk.dm-1.out‚2834.927734375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.82700443267822) disk./.usagePercent‚76.46193610711433& disk.nvme0n1.out‚35570.45361328125 disk.nvme1c1n1.in‚3.25 disk.md0.in‚0( proc.memory.availableMB‚ 192.18359375* proc.memory.percent‚17.522621154785156¡Åå`›È‚(: ¶¡õÒÊ­ô+ gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.19566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03lB~2É- _wandbruntime‚3491  _runtime‚3491ÙfÌúÆÊÀ trainer/global_step‚8699 $ _timestamp‚1.784500417911545e+09 _runtime‚3491.822457822 _step‚702 & lr-AdamW/pg1‚9.192477422426331e-05 & lr-AdamW/pg2‚9.192477422426331e-05á#üBËËÅ $ _timestamp‚1.784500417911545e+09 _runtime‚3491.822457822 _step‚702 & lr-AdamW/pg1‚9.192477422426331e-05 & lr-AdamW/pg2‚9.192477422426331e-05 trainer/global_step‚8699¾¯õé¡2Ì- _wandbruntime‚3491  _runtime‚34918ŽKKuÍÒO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319818636055552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340933180070400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326106892440576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66071369 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6090628246429 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340933075009024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22316561 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326106695512064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88114723 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340933180070400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38265987 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336571672771827 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚334823432055808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6109642953216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154186092 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66071367 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336571775959661 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8699 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60582548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154175947 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 WJDeviceStatsMonitor,Œäc.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚334823432055808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66071367 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6109642953216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334823449095168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 % _timestamp‚1.7845004179136856e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88114723 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6109730975232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60582302 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6288256385024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88104580 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6109730975232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22316318 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6288084360192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340933075009024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330481130673872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88104580 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0f1Ý'Î×O TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38265984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66071367 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336571775959661 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8699 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60582548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154175947 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚334823432055808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66071367 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6109642953216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334823449095168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 % _timestamp‚1.7845004179136856e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88114723 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6109730975232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60582302 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6288256385024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88104580 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6109730975232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22316318 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6288084360192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340933075009024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330481130673872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88104580 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319818636055552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340933180070400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326106892440576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66071369 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6090628246429 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340933075009024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22316561 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326106695512064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88114723 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340933180070400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38265987 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336571672771827 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚334823432055808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6109642953216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154186092 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336609323978977 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336609220791031 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22319016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66078690 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6110321062912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6091218049399 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38270274 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22318755 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334860815462400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326143293171200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6288961958400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚334860798423040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340971224554496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6110409092096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88124635 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚334860815462400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88124635 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚334860798423040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330518002741632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330518019780992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154193166 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66078688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66078690 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60589029 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154203325 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88114478 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104¸ð¼‰!ÑáM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340971224554496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6110409092096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88124635 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚334860815462400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88124635 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚334860798423040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330518002741632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330518019780992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154193166 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66078688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66078690 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60589029 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154203325 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88114478 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3492.075329016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6110409092096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88114478 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336609323978977 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336609220791031 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22319016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66078690 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6110321062912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6091218049399 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38270274 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚}(?^22318755 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334860815462400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326143293171200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6288961958400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚334860798423040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208ÀT¶#;2Ò- _wandbruntime‚3492  _runtime‚3492 ùÝ}Ó‚ 0loss/FRt2/assignment_loss‚1.0554289547902234 2loss/SRqqt1/assignment_loss‚0.7796797483020321 2loss/SRqqt2/assignment_loss‚0.6599108900775045 /loss/FRt1/detection_loss‚0.5596024167457528 trainer/global_step‚8699 2loss/FBt1/assignment_loss‚0.005376769739383137 2loss/FBt2/assignment_loss‚0.004637173491310356 /loss/FRt2/detection_loss‚1.4588120331706584 1loss/SRqqt1/detection_loss‚1.1070628901943564 &loss/total_loss‚10.073339045876988 % _timestamp‚1.7845004181655295e+09 0loss/SRqqt2/detection_loss‚0.660933245685725 .loss/FBt1/detection_loss‚0.720203709056867 /loss/FBt2/detection_loss‚0.8022277680021234 epoch‚10 _runtime‚3492.076186187 /loss/FRt1/assignment_loss‚2.259463446621051 _step‚705ás·@ Ô‡ /loss/FBt2/detection_loss‚0.8022277680021234 epoch‚10 _runtime‚3492.076186187 /loss/FRt1/assignment_loss‚2.259463446621051 _step‚705 0loss/FRt2/assignment_loss‚1.0554289547902234 2loss/SRqqt1/assignment_loss‚0.7796797483020321 2loss/SRqqt2/assignment_loss‚0.6599108900775045 /loss/FRt1/detection_loss‚0.5596024167457528 trainer/global_step‚8699 2loss/FBt1/assignment_loss‚0.005376769739383137 2loss/FBt2/assignment_loss‚0.004637173491310356 /loss/FRt2/detection_loss‚1.4588120331706584 1loss/SRqqt1/detection_loss‚1.1070628901943564 &loss/total_loss‚10.073339045876988 % _timestamp‚1.7845004181655295e+09 0loss/SRqqt2/detection_loss‚0.660933245685725 .loss/FBt1/detection_loss‚0.720203709056867Á×0peiÕ‚(:Þ Å¡õÒÊ׌, disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375& disk.nvme0n1.out‚35579.63720703125# memory_percent‚98.8274097442627' proc.memory.availableMB‚ 192.1171875 proc.memory.rssMB‚ 2870.90625 network.sent‚ 1.1568443e+07 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 19.68359375 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 1.02718e+06% disk./.usageGB‚1173.8783073425293 disk.md0.in‚0* proc.memory.percent‚17.522621154785156 cpu‚26.305032782749294 disk.dm-1.in‚0 disk.dm-1.out‚2835.029296875) disk./.usagePercent‚76.46170378391332 disk.nvme1c1n1.in‚3.25' disk.nvme1c1n1.out‚15375.8662109375% disk.nvme1n1.out‚15375.8662109375 disk.md0.out‚ 103.22265625( disk.nvme0c0n1.out‚35579.63720703125# disk.dm-0.out‚47900.05029296875"¤ šÖ‚(: Å¡õÒ—ÚŸ+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.268' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.089333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 gׂ(:Ü Ô¡õÒ¼øÝ+% disk./.usageGB‚1173.8738327026367# memory_percent‚99.2218017578125* proc.memory.percent‚17.522621154785156 cpu‚26.303881726723013 proc.cpu.threads‚17 network.sent‚ 1.1619662e+07& disk.nvme0n1.out‚35584.79736328125 disk.md0.out‚ 103.5234375! disk.nvme0c0n1.in‚ 16.43359375( disk.nvme0c0n1.out‚35584.79736328125! proc.memory.availableMB‚127.5 disk.nvme0n1.in‚ 16.43359375( disk.nvme1c1n1.out‚15383.01904296875& disk.nvme1n1.out‚15383.01904296875 disk.md0.in‚0# disk.dm-0.out‚47911.54248046875 disk.rbd0.in‚0 proc.memory.rssMB‚ 2870.90625 network.recv‚ 1.031506e+06) disk./.usagePercent‚76.46141232389752 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0" disk.dm-1.out‚2835.19384765625 disk.rbd0.out‚0wl˜=Ø‚(:„ Ô¡õÒãÅŠ, gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.075' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç=<2Ù- _wandbruntime‚3511  _runtime‚3511ËLœÇÚÁ % _timestamp‚1.7845004375298865e+09 _runtime‚3511.441501724 _step‚706 & lr-AdamW/pg1‚9.180412440956706e-05 & lr-AdamW/pg2‚9.180412440956706e-05 trainer/global_step‚87498ÄÕ{ÌÛÆ trainer/global_step‚8749 % _timestamp‚1.7845004375298865e+09 _runtime‚3511.441501724 _step‚706 & lr-AdamW/pg1‚9.180412440956706e-05 & lr-AdamW/pg2‚9.180412440956706e-05Â[¤—Â2Ü- _wandbruntime‚3511  _runtime‚3511¸]Ü¢×'ÝÑO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚342835436173824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6143668717056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6124371573363 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88601332 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22441280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3511.441559584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342835331112448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155048894 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88611475 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6323295205376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336691750417408 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338449105635235 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338449208823069 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6143580695040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66437419 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38480484 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6143580695040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚342835436173824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38480484 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6143580695040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚342835436173824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342835331112448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155048894 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88611475 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6323295205376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Ã/Óˆ2ß- _wandbruntime‚3511  _runtime‚3511¬€6àÜM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66444740 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6144346833920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66444738 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338486756842385 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338486653654439 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6125133673393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22443727 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155055968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332361623168992 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336729116784640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155066127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6323805868544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321639089075200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7845004378009398e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332361606129632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6125047524807 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88611230 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.a+Á0sÌllocated‚336729133824000 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60928501 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155055968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60928762 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155066127 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336729116784640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6144258804736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6144346833920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38484774 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336729133824000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66444740?c+ç&ááM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321639089075200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7845004378009398e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332361606129632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6125047524807 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88611230 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336729133824000 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60928501 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155055968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60928762 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155066127 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336729116784640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6144258804736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6144346833920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38484774 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336729133824000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66444740 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66444740 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6144346833920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66444738 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338486756842385 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338486653654439 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6125133673393 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22443727 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155055968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332361623168992 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336729116784640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155066127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6323805868544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ÄkEó„2â- _wandbruntime‚3511  _runtime‚3511Lý‰ã€ /loss/FRt2/assignment_loss‚1.051224766458784 0loss/FBt1/assignment_loss‚0.0733128654012063 3loss/FBt2/assignment_loss‚0.0012116197642747645 /loss/FRt1/detection_loss‚0.5828169626727251 %loss/total_loss‚9.854186750427539 trainer/global_step‚8749 /loss/FRt2/detection_loss‚1.4244185451301745 /loss/FBt1/detection_loss‚0.7037987180519849 % _timestamp‚1.7845004378020236e+09 _runtime‚ 3511.71271483 0loss/FRt1/assignment_loss‚2.2496752022905686 /loss/FBt2/detection_loss‚0.5533430849614202 _step‚709 2loss/SRqqt1/assignment_loss‚0.8153643050168488 2loss/SRqqt2/assignment_loss‚0.5992852648187109 0loss/SRqqt1/detection_loss‚1.209558458331583 1loss/SRqqt2/detection_loss‚0.5901769575292581 epoch‚10‡FFf ä… 2loss/SRqqt1/assignment_loss‚0.8153643050168488 2loss/SRqqt2/assignment_loss‚0.5992852648187109 0loss/SRqqt1/detection_loss‚1.209558458331583 1loss/SRqqt2/detection_loss‚0.5901769575292581 epoch‚10 /loss/FRt2/assignment_loss‚1.051224766458784 0loss/FBt1/assignment_loss‚0.0733128654012063 3loss/FBt2/assignment_loss‚0.0012116197642747645 /loss/FRt1/detection_loss‚0.5828169626727251 %loss/total_loss‚9.854186750427539 trainer/global_step‚8749 /loss/FRt2/detection_loss‚1.4244185451301745 /loss/FBt1/detection_loss‚0.7037987180519849 % _timestamp‚1.7845004378020236e+09 _runtime‚ 3511.71271483 0loss/FRt1/assignment_loss‚2.2496752022905686 /loss/FBt2/detection_loss‚0.5533430849614202 _step‚709ÅlÎ9iå‚(:Þ ã¡õÒ•¹Ÿ+ disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚35594.3955078125' disk.nvme1c1n1.out‚15414.6337890625 proc.memory.rssMB‚ 2870.90625 proc.cpu.threads‚17 network.recv‚ 1.034909e+06% disk./.usageGB‚1172.4695472717285# disk.dm-0.out‚47951.85791015625" disk.dm-1.out‚2835.27587890625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.43359375 disk.nvme1n1.in‚3.25% disk.nvme1n1.out‚15414.6337890625) disk./.usagePercent‚76.36994282831647 disk.dm-0.in‚ 19.68359375 disk.md0.in‚0 disk.md0.out‚ 103.90625( proc.memory.availableMB‚ 127.46484375 cpu‚26.302106392119924 network.sent‚ 1.1622584e+07! disk.nvme0c0n1.in‚ 16.43359375% disk.nvme0n1.out‚35594.3955078125 disk.nvme1c1n1.in‚3.25$ memory_percent‚99.22201633453369* proc.memory.percent‚17.522621154785156y²É$šæ‚(: ã¡õÒÐø+ gpu.0.gpu‚54.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚196.508' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.50266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ª2ç- _wandbruntime‚3531  _runtime‚3531ÎYŒ²Åè¿ % _timestamp‚1.7845004575376518e+09 _runtime‚3531.449403376 _step‚710 % lr-AdamW/pg1‚9.16828190913551e-05 % lr-AdamW/pg2‚9.16828190913551e-05 trainer/global_step‚8799¤’û»ÊéÄ % lr-AdamW/pg2‚9.16828190913551e-05 trainer/global_step‚8799 % _timestamp‚1.7845004575376518e+09 _runtime‚3531.449403376 _step‚710 % lr-AdamW/pg1‚9.16828190913551e-05ÆŠKæó2ê- _wandbruntime‚3531  _runtime‚3531ÒÑ#Ø'ëÒO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155901551 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340326641686477 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66803467 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚338560068779008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155911696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61260986 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334168337449872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6177606458880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344737587215872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6177606458880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38694984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338560068779008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚344737692277248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338560085818368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22566254 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22566002 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329746409055232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6158201048771 % _timestamp‚1.7845004575383286e+09 _runtime‚3531.449466766 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6177518436864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 _step‚711 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338560085818368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344737587215872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66803469 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340326538498643 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89098084 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6177518436864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323388151780352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89108227 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329746195349504 trainer/global_step‚8799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38694987 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6158287197245 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66803469 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66803467 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6358068472832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚334168354489232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61261241 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323388126876672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6358257274880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89098084 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344737692277248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344737587215872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6177606458880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38694984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338560068779008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚344737692277248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338560085818368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22566254 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22566002 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329746409055232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6158201048771 % _timestamp‚1.7845004575383286e+09 _runtime‚3531.449466766 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6177518436864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 _step‚711 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338560085818368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344737587215872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66803469 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340326538498643 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89098084 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6177518436864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323388151780352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 W“HüIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89108227 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329746195349504 trainer/global_step‚8799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38694987 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6158287197245 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66803469 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66803467 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6358068472832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚334168354489232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61261241 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323388126876672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6358257274880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89098084 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344737692277248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89118139 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚8799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚338597452185600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323423846937600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89107982 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6178284575744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚338597452185600 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚344775736761344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚338597435146240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344775631692800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66810788 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89107982 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61267966 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61267715 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66810788 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344775736761344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334205226556992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340364189705793 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6178196546560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155918770 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66810790 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6178284575744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22568441 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6178196546560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66810790 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155928929 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155928929 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ÿšç&ïáM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340364189705793 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6178196546560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155918770 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66810790 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6178284575744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22568441 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6178196546560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66810790 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155928929 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155928929 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334205209517632 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6358953968128 % _timestamp‚1.7845004578053086e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155918770 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22568689 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329782800905728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6158963148801 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚338597435146240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89118139 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚8799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚338597452185600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323423846937600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89107982 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6178284575744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚338597452185600 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚344775736761344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚338597435146240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344775631692800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66810788 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89107982 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61267966 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61267715 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66810788 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344775736761344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334205226556992ÈŽ¢2ð- _wandbruntime‚3531  _runtime‚3531«Ÿ‚, ñƒ 2loss/SRqqt1/assignment_loss‚0.6111019858591524 2loss/SRqqt2/assignment_loss‚0.6199976459469377 1loss/FBt1/assignment_loss‚0.12880020820978189 2loss/FBt2/assignment_loss‚0.000629002077497201 0loss/FBt2/detection_loss‚0.37100374000802844 %loss/total_loss‚9.452541551806661 _runtime‚3531.716933399 _step‚713 0loss/FRt1/assignment_loss‚2.0625594770592732 /loss/FRt1/detection_loss‚0.5301542193258975 /loss/FRt2/detection_loss‚1.4722549670322005 1loss/SRqqt1/detection_loss‚1.0211406365119327 1loss/SRqqt2/detection_loss‚0.6026185002033664 epoch‚10 trainer/global_step‚8799 0loss/FRt2/assignment_loss‚0.9992519969289954 /loss/FBt1/detection_loss‚1.0330291726435958 $ _timestamp‚1.784500457806317e+09ä5…òˆ 0loss/FBt2/detection_loss‚0.37100374000802844 %loss/total_loss‚9.452541551806661 _runtime‚3531.716933399 _step‚713 0loss/FRt1/assignment_loss‚2.0625594770592732 /loss/FRt1/detection_loss‚0.5301542193258975 /loss/FRt2/detection_loss‚1.4722549670322005 1loss/SRqqt1/detection_loss‚1.0211406365119327 1loss/SRqqt2/detection_loss‚0.6026185002033664 epoch‚10 trainer/global_step‚8799 0loss/FRt2/assignment_loss‚0.9992519969289954 /loss/FBt1/detection_loss‚1.0330291726435958 $ _timestamp‚1.784500457806317e+09 2loss/SRqqt1/assignment_loss‚0.6111019858591524 2loss/SRqqt2/assignment_loss‚0.6199976459469377 1loss/FBt1/assignment_loss‚0.12880020820978189 2loss/FBt2/assignment_loss‚0.000629002077497201ÉåÃ1ô_ó‚(:Ô ò¡õÒÚÙî+# disk.nvme1n1.out‚17156.23046875 disk.md0.in‚0 disk.dm-0.out‚ 51476.5390625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚37378.5087890625) proc.memory.percent‚17.53067970275879 cpu‚26.3038469147845 network.sent‚ 1.1699017e+07 disk.dm-0.in‚ 19.68359375! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 network.recv‚ 1.038969e+06) disk./.usagePercent‚76.43080454970625 disk.md0.out‚ 104.296875% disk.nvme1c1n1.out‚17156.23046875" proc.memory.rssMB‚ 2872.2265625 proc.cpu.threads‚17 disk.nvme1n1.in‚3.25! disk.dm-1.out‚2835.4716796875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 16.43359375% disk.nvme0n1.out‚37378.5087890625$ memory_percent‚99.22823905944824' proc.memory.availableMB‚ 126.4453125% disk./.usageGB‚1173.4039268493652Œô$šô‚(: ò¡õÒýÁ, gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚114.896' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.29866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R™³p2õ- _wandbruntime‚3551  _runtime‚3551_¾FHö¼{ :"Purity/0FRt2SRqqt0FBt/event_purity‚0.42527303099632263 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09573765844106674 particle/accuracy_2_of_4‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15000000596046448 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6825319528579712 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027045542374253273 ,jet/accuracy_4_of_4‚0.014682539738714695 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.723931610584259 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820460081100464 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938985705375671 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 )loss/val_total_loss‚9.107399541377115 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9938756823539734 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6253914833068848 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.615946888923645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03183668106794357 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 epoch‚10 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.166657492518425 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41317886114120483 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03625449538230896 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9141233563423157 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5352923274040222 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5518375039100647 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_3_of_3‚0.0719568058848381 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03310065716505051 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2580731511116028 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3672839403152466 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20053286850452423 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18596284091472626 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922360777854919 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7305619716644287 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 _step‚714 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6047390699386597 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994796633720398 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7345324754714966 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 jet/accuracy_3_of_5‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.714919924736023 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14406931400299072 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2964041233062744 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5091146230697632 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47977393865585327 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015827583149075508 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.993829071521759 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8052842020988464 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08530642092227936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43508294224739075 ,} ©z*&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1804492175579071 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46719983220100403 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04405701160430908 jet/accuracy_4_of_5‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8585730791091919 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3682253360748291 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3656248450279236 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03823501989245415 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7407533526420593 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7090024948120117 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6203140020370483 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18791668117046356 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6391276121139526 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8082706332206726 +jet/accuracy_2_of_2‚0.14562590420246124 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8072207570075989 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19006043672561646 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16725322604179382 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.036045610904693604 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9658068418502808 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969671368598938 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7260308861732483 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15308555960655212 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6008221507072449 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 (particle/accuracy‚0.8513686656951904 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5393378138542175 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017160743474960327 /particle/accuracy_4_of_4‚0.9629918932914734 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 *jet/accuracy_1_of_3‚0.9152635931968689 7validation_average_jet_accuracy‚0.35298794507980347 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_5‚1 _runtime‚3551.878131711 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6644429564476013 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027045542374253273 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8539139032363892 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18758004903793335 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6497884392738342 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7519564628601074 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5849902629852295 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6205388307571411 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16253037750720978 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.813400387763977 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_5_of_5‚0 'particle/f_score‚0.6486769318580627 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333730697632 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9829035401344299 *jet/accuracy_1_of_1‚0.3099620044231415 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6742573976516724 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6650570631027222 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.997012197971344 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.0886337161064148 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15491308271884918 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2004319131374359 particle/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6427268981933594 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6035862565040588 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6045247316360474 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15423622727394104 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17121943831443787 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8585730791091919 *jet/accuracy_2_of_4‚0.8128939270973206 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9086575508117676 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6149723529815674 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5429404973983765 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_2‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995014488697052 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9791627526283264 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03780686482787132 *jet/accuracy_1_of_2‚0.7533619999885559 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2539886236190796 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7626374363899231 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7995613217353821 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10324960947036743 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6685342788696289 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6366429328918457 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_5_of_5‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008322762325406075 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998765230178833 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7294492721557617 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7615715265274048 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.81025630235672 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16230085492134094 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8572203516960144 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5396862626075745 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_4‚0.9594181776046753 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6790229082107544 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19924204051494598 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856334924697876 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14359787106513977 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18791668117046356 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2905672490596771 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7461062669754028 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6065089106559753 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 % _timestamp‚1.7845004779667597e+09 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 /particle/accuracy_3_of_4‚0.9981684684753418 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19119882583618164 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8976864814758301 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9986110925674438 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6205388307571411 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07132704555988312 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5474534034729004 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.442943811416626 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7907894253730774 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954930543899536 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1837940663099289 +particle/specificity‚0.9094078540802002 trainer/global_step‚8810Ä”ŽïÇ=÷Á{ 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995014488697052 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9791627526283264 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03780686482787132 *jet/accuracy_1_of_2‚0.7533619999885559 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2539886236190796 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7626374363899231 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7995613217353821 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10324960947036743 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6685342788696289 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6366429328918457 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_5_of_5‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008322762325406075 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998765230178833 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7294492721557617 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7615715265274048 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.81025630235672 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16230085492134094 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8572203516960144 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5396862626075745 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_4‚0.9594181776046753 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6790229082107544 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19924204051494598 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856334924697876 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14359787106513977 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18791668117046356 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2905672490596771 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7461062669754028 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6065089106559753 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 % _timestamp‚1.7845004779667597e+09 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 /particle/accuracy_3_of_4‚0.9981684684753418 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19119882583618164 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8976864814758301 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9986110925674438 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6205388307571411 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07132704555988312 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5474534034729004 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.442943811416626 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7907894253730774 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954930543899536 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1837940663099289 +particle/specificity‚0.9094078540802002 trainer/global_step‚8810 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.42527303099632263 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09573765844106674 particle/accuracy_2_of_4‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15000000596046448 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6825319528579712 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027045542374253273 ,jet/accuracy_4_of_4‚0.014682539738714695 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.723931610584259 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820460081100464 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938985705375671 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 )loss/val_total_loss‚9.107399541377115 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9938756823539734 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6253914833068848 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.615946888923645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03183668106794357 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 epoch‚10 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.166657492518425 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41317886114120483 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03625449538230896 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9141233563423157 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5352923274040222 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5518375039100647 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_3_of_3‚0.0719568058848381 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03310065716505051 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2580731511116028 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3672839403152466 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20053286850452423 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18596284091472626 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922360777854919 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7305619716644287 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 _step‚714 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6047390699386597 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994796633720398 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7345324754714966 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 jet/accuracy_3_of_5‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.714919924736023 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14406931400299072 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2964041233062744 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5091146230697632 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47977393865585327 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015827583149075508 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.993829071521759 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8052842020988464 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08530642092227936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43508294224739075 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1804492175579071 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46719983220100403 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04405701160430908 jet/accuracy_4_of_5‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8585730791091919 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3682253360748291 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3656248450279236 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03823501989245415 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7407533526420593 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7090024948120117 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6203140020370483 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18791668117046356 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6391276121139526 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8082706332206726 +jet/accuracy_2_of_2‚0.14562590420246124 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8072207570075989 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19006043672561646 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16725322604179382 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.036045610904693604 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9658068418502808 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969671368598938 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7260308861732483 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15308555960655212 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6008221507072449 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 (particle/accuracy‚0.8513686656951904 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5393378138542175 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017160743474960327 /particle/accuracy_4_of_4‚0.9629918932914734 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 *jet/accuracy_1_of_3‚0.9152635931968689 7validation_average_jet_accuracy‚0.35298794507980347 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_5‚1 _runtime‚3551.878131711 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6644429564476013 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9973545074462891 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027045542374253273 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8539139032363892 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18758004903793335 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6497884392738342 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7519564628601074 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5849902629852295 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6205388307571411 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16253037750720978 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.813400387763977 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_5_of_5‚0 'particle/f_score‚0.6486769318580627 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333730697632 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9829035401344299 *jet/accuracy_1_of_1‚0.3099620044231415 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6742573976516724 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6650570631027222 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.997012197971344 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.0886337161064148 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9107142686843872 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15491308271884918 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2004319131374359 particle/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6427268981933594 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6035862565040588 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6045247316360474 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15423622727394104 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17121943831443787 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8585730791091919 *jet/accuracy_2_of_4‚0.8128939270973206 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9086575508117676 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6149723529815674 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5429404973983765 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_2‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929Ê»y2^jø‚(:ß ¢õÒÍÔó+ disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚2899.3857421875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375& proc.memory.availableMB‚ 192.953125 network.recv‚ 1.043525e+06 disk.nvme1n1.in‚3.25 disk.md0.out‚ 104.71875" disk.dm-0.out‚51499.1298828125 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚17224.04443359375& disk.nvme1n1.out‚17224.04443359375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.43359375& disk.nvme0n1.out‚37398.11669921875 cpu‚26.312728431041165 network.sent‚ 1.176017e+07) disk./.usagePercent‚76.43095910054157 disk.dm-0.in‚ 19.68359375( disk.nvme0c0n1.out‚37398.11669921875$ memory_percent‚98.82230758666992# proc.memory.rssMB‚ 2875.52734375) proc.memory.percent‚17.55082607269287 proc.cpu.threads‚17% disk./.usageGB‚1173.4062995910645OÎJŽšù‚(: ¢õÒ®¾, gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚138.628' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚46.20933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wÜMö2ú- _wandbruntime‚3567  _runtime‚3567 I uÇûÁ & lr-AdamW/pg1‚9.156086021257839e-05 & lr-AdamW/pg2‚9.156086021257839e-05 trainer/global_step‚8849 % _timestamp‚1.7845004936259806e+09 _runtime‚3567.538416431 _step‚715zËÖÌüÆ & lr-AdamW/pg1‚9.156086021257839e-05 & lr-AdamW/pg2‚9.156086021257839e-05 trainer/global_step‚8849 % _timestamp‚1.7845004936259806e+09 _runtime‚3567.538416431 _step‚715ËŒ'ì2ý- _wandbruntime‚3567  _runtime‚3567bw…þÒO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89762102 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341401067774976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22768166 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67406985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6231876495872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67406987 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6413703680000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347632927231488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61771473 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343192958025803 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6212510680219 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326098332906496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.alloca¶ìðzSted‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347633032292864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39003564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341401067774976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845004936266987e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341401050735616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67406985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89772245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67406987 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231964517888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚336980447345584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347632927231488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347633032292864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61771733 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6231876495872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231964517888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341401050735616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326098308002816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚716 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3116827519ÍrÝ'ÿ×O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89772245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67406987 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231964517888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚336980447345584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347632927231488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347633032292864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61771733 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6231876495872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231964517888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341401050735616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326098308002816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚716 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89762102 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341401067774976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22768166 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67406985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6231876495872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67406987 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6413703680000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347632927231488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61771473 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343192958025803 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6212510680219 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326098332906496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347633032292864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39003564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341401067774976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845004936266987e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341401050735616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67406985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Ì#Ò2€- _wandbruntime‚3567  _runtime‚3567jí·â&ÜM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39007854 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337017319413344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326134028063744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347671076776960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341438417102848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232642634752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 trainer/global_step‚8849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22770671 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157196465 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6414216174080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326134003160064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6414392386048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67414306 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347670971708416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157186306 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67414306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6213100483189 _runtime‚3567.802326283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6213186631775 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67414308 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61778528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚717 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157186306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22770415 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6232554605568 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61778269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337017302373984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6232554605568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89772000 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157196465W…Ñ™‚áM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232642634752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89772000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61778269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337017302373984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6232554605568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89772000 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157196465 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39007854 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337017319413344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326134028063744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347671076776960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341438417102848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232642634752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 trainer/global_step‚8849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22770671 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.al@©¦ÀN located‚ 157196465 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6414216174080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326134003160064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6414392386048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67414306 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347670971708416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157186306 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67414306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6213100483189 _runtime‚3567.802326283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6213186631775 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67414308 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61778528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚717 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157186306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22770415 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6232554605568 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349535183334912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343269369097216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345070390889211 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338824050733584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚345070287701377 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚3586.749386998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349535288396288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158031889 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6265902259712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334331729336832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚8899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327883090768896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67773037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6246340155627 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22891667 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6246254007153 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90268997 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62109731 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343269386136576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67773035 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343269386136576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6448638567936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334331517728256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349535183334912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343269369097216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345070390889211 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338824050733584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚345070287701377 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚3586.749386998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349535288396288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158031889 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6265902259712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334331729336832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚8899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327883090768896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67773037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6246340155627 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22891667 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6246254007153 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90268997 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62109731 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343269386136576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67773035 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343269386136576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6448638567936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334331517728256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751Ð`áô2- _wandbruntime‚3587  _runtime‚3587_÷ò$‘ÛM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚8899 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158059267 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343306752503808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6266492347392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62116458 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158059267 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67780356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349573332880384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349573332880384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6266580376576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334368124451840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343306735464448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67780358 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349573227811840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343306752503808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62116723 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22894104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6246929958597 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158049108 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67780358 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6266492347392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6247016107183 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90278909 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚345107835720581 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327918785926144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚721 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90278909 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90268752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784500513093404e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327918761022464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338860905761984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487Ô—Vcæ&’àM PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343306735464448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67780358 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349573227811840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343306752503808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62116723 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22894104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6246929958597 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158049108 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67780358 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6266492347392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6247016107183 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90278909 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚345107835720581 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327918785926144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚721 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90278909 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90268752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784500513093404e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327918761022464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338860905761984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚8899 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158059267 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343306752503808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6266492347392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62116458 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158059267 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67780356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349573332880384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349573332880384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6266580376576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334368124451840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ñ–mM2“- _wandbruntime‚3587  _runtime‚3587ß;X ”ƒ _runtime‚3587.005034274 _step‚722 0loss/FRt1/assignment_loss‚2.1460512496982087 .loss/FBt2/detection_loss‚0.379392801044834 2loss/SRqqt2/assignment_loss‚0.6043773085578659 3loss/FBt2/assignment_loss‚0.0007661246734719787 /loss/FRt2/detection_loss‚1.5010126900023466 % _timestamp‚1.7845005130943391e+09 /loss/FRt1/detection_loss‚0.5528053876763491 0loss/SRqqt2/detection_loss‚0.614691683622874 /loss/FBt1/detection_loss‚0.8676072973267812 &loss/total_loss‚10.011218144624635 0loss/FRt2/assignment_loss‚1.1410082954870884 2loss/SRqqt1/assignment_loss‚0.6957533839233476 2loss/FBt1/assignment_loss‚0.029135705220943765 0loss/SRqqt1/detection_loss‚1.478616217390523 epoch‚11 trainer/global_step‚8899‹®ì•ˆ .loss/FBt2/detection_loss‚0.379392801044834 2loss/SRqqt2/assignment_loss‚0.6043773085578659 3loss/FBt2/assignment_loss‚0.0007661246734719787 /loss/FRt2/detection_loss‚1.5010126900023466 % _timestamp‚1.7845005130943391e+09 /loss/FRt1/detection_loss‚0.5528053876763491 0loss/SRqqt2/detection_loss‚0.614691683622874 /loss/FBt1/detection_loss‚0.8676072973267812 &loss/total_loss‚10.011218144624635 0loss/FRt2/assignment_loss‚1.1410082954870884 2loss/SRqqt1/assignment_loss‚0.6957533839233476 2loss/FBt1/assignment_loss‚0.029135705220943765 0loss/SRqqt1/detection_loss‚1.478616217390523 epoch‚11 trainer/global_step‚8899 _runtime‚3587.005034274 _step‚722 0loss/FRt1/assignment_loss‚2.1460512496982087Ò  Æµj–‚(:ß ®¢õÒ’·ë+ network.recv‚ 1.062453e+06& disk.nvme0n1.out‚37814.71044921875 disk.md0.in‚0 disk.dm-1.out‚2963.794921875$ memory_percent‚98.82352352142334) proc.memory.percent‚17.55082607269287 disk.nvme1n1.in‚3.25( disk./.usagePercent‚76.3699033209486 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚37814.71044921875 disk.nvme1c1n1.in‚3.25' disk.nvme1c1n1.out‚17494.2353515625 cpu‚26.308834698908008 network.sent‚ 1.1889159e+07! disk.nvme0c0n1.in‚ 16.43359375% disk.nvme1n1.out‚17494.2353515625 disk.md0.out‚ 106.28125# disk.dm-0.out‚52111.68408203125( proc.memory.availableMB‚ 192.75390625# proc.memory.rssMB‚ 2875.52734375 proc.cpu.threads‚17% disk./.usageGB‚1172.4689407348633 disk.nvme0n1.in‚ 16.43359375 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0çz ì—‚(:… ®¢õÒݦ–, gpu.0.gpu‚91.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.612' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ ÆT2˜- _wandbruntime‚3605  _runtime‚36058 ]Ç™Á _runtime‚3605.647562597 _step‚723 & lr-AdamW/pg1‚9.131498959744328e-05 & lr-AdamW/pg2‚9.131498959744328e-05 trainer/global_step‚8949 % _timestamp‚1.7845005317344115e+09ˆ(m¨ÌšÆ _step‚723 & lr-AdamW/pg1‚9.131498959744328e-05 & lr-AdamW/pg2‚9.131498959744328e-05 trainer/global_step‚8949 % _timestamp‚1.7845005317344115e+09 _runtime‚3605.647562597ÓÍ|_2›- _wandbruntime‚3605  _runtime‚3605”§Ø'œÒO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39432567 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346947823752619 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845005317350101e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345137687458816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90765749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345137687458816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90755606 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90765749 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62447100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6483544093696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6280169631035 trainer/global_step‚8949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340667654121584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158904836 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68139085 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90755606 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚3605.647624537 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚351437544499712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6299840001536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23014536 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351437439438336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351437544499712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚345137704498176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6483369971712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68139087 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6299751979520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6299751979520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336151193699328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351437439438336 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329667848631296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329667823727616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68139085 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6299840001536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362}. eð ×O \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340667637082224 _step‚724 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158894691 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351437439438336 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329667848631296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329667823727616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68139085 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6299840001536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39432567 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346947823752619 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845005317350101e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345137687458816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90765749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345137687458816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90755606 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚äJì„í0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90765749 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62447100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6483544093696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6280169631035 trainer/global_step‚8949 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340667654121584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158904836 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68139085 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90755606 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚3605.647624537 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚351437544499712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6299840001536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23014536 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351437439438336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351437544499712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚345137704498176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6483369971712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68139087 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6299751979520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6299751979520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336151193699328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62453860 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336187581129216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340704509149984 _runtime‚3605.903486408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68146408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784500531992141e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158922069 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90775661 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6484251039232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90775661 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329703543788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6484062244352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158922069 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23017259 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329703518884864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90765504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6300518118400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336187794827776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6280845582591 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345175053826048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346985268583989 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152nS|öæ& àM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158911910 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158922069 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23017259 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329703518884864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90765504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6300518118400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336187794827776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6280845582591 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345175053826048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346985268583989 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90765504 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351475588983808 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62453860 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336187581129216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340704509149984 _runtime‚3605.903486408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68146408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784500531992141e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158922069 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90775661 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6484251039232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90775661 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329703543788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6484062244352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353339695541760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6333777743360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62784411 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6313999106443 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353339800603136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23137590 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 693737fÊû”P8816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353339695541760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91262501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348825256616027 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚342511240470224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91252358 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337971093885440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39647067 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331452606493696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23137347 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845005505988898e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348825153428193 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353339800603136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347006005820416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6333777743360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68505135 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68505137 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331452581590016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348825153428193 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353339800603136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347006005820416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6333777743360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68505135 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68505137 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331452581590016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353339695541760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6333777743360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62784411 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6313999106443 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353339800603136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23137590 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353339695541760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91262501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348825256616027 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚342511240470224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145Øß·²2¬- _wandbruntime‚3624  _runtime‚3624ö|¦’â&­ÜM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6334455860224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68512458 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68512456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91272413 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6314588909413 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159774712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6334367831040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331488276747264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338007270704128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331488301650944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348862804635343 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚342548112537984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159774712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚342548129577344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159784871 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353377740018688 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347043389227008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159784871 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68512458 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6334367831040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6518993956864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚3624.766150475 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348862701447397 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6314675057999 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚8999 % _timestamp‚1.7845005508550265e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39651357 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62791121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚729 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39651354 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91272413 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚338007482305536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91262256 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347043389227008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6334455860224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91262256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347043372187648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0O]Üœ®áM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚729 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39651354 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91272413 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚338007482305536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91262256 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347043389227008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6334455860224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91262256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347043372187648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6334455860224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68512458 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68512456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91272413 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6314588909413 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159774712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6334367831040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331488276747264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338007270704128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331488301650944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348862804635343 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚342548112537984 VGDeviceStatsMonitor.on_train_basþfÊKtch_end/reserved_bytes.large_pool.current‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159774712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚342548129577344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159784871 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353377740018688 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347043389227008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159784871 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68512458 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6334367831040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6518993956864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚3624.766150475 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348862701447397 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6314675057999 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚8999 % _timestamp‚1.7845005508550265e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39651357 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62791121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23260572 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344354860897584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91759253 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333237364356096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355242056706560 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23260325 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚732 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348874324182016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63122139 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355241951645184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68871185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348874324182016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63121889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344354843858224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91749110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39861564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68871187 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6553441196544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6367715485184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348874341221376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367627463168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350702586291601 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68871187 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339790805552640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333237339452416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 _runtime‚3643.093868399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350702689479435 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23260572 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344354860897584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91759253 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333237364356096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355242056706560 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23260325 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚732 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348874324182016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63122139 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355241951645184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68871185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348874324182016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63121889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344354843858224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91749110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39861564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68871187 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6553441196544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6367715485184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348874341221376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367627463168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350702586291601 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68871187 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339790805552640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333237339452416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 _runtime‚3643.093868399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350702689479435 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3643.353817129 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348911690549248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355279996122112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348911707588608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39865857 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23263046 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6348504533407 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23262792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845005694427361e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.allPÝò·.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344391732965344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339827002783744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355280101190656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6368393602048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68878508 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91769165 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91759008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348911707588608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68878508 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368305572864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63128646 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355280101190656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _step‚733 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68878506 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333273059513344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348911690549248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333273034609664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350740134310805 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚344391715925984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160647673 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6368393602048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160647673 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63128903 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330³7%ç&¼áM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160647673 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6368393602048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160647673 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63128903 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91759008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6348418384821 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368305572864 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3643.353817129 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348911690549248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355279996122112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348911707588608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39865857 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23263046 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6348504533407 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23262792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845005694427361e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344391732965344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339827002783744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355280101190656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6368393602048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68878508 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91769165 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91759008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348911707588608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68878508 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368305572864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63128646 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355280101190656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _step‚733 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68878506 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333273059513344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348911690549248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333273034609664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350740134310805 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚344391715925984ÝÁÕ¶z2½- _wandbruntime‚3643  _runtime‚3643ùì£v ¾ƒ 2loss/SRqqt1/assignment_loss‚0.7468756970544285 2loss/SRqqt2/assignment_loss‚0.6069211800519048 3loss/FBt1/assignment_loss‚0.0006340627719888895 3loss/FBt2/assignment_loss‚0.0008553805256725559 % _timestamp‚1.7845005694436696e+09 /loss/FRt1/assignment_loss‚2.209143015064395 /loss/FRt2/detection_loss‚1.5402555226484158 /loss/FBt1/detection_loss‚1.7909481786191463 /loss/FBt2/detection_loss‚0.7311830056612106 epoch‚11 _runtime‚ 3643.35432004 _step‚734 /loss/FRt2/assignment_loss‚1.112395479530096 1loss/SRqqt1/detection_loss‚1.1411754661542062 trainer/global_step‚9049 /loss/FRt1/detection_loss‚0.5863789854873004 0loss/SRqqt2/detection_loss‚0.678233862103453 &loss/total_loss‚11.144999835672218Ÿ¨ßí¿ˆ 2loss/SRqqt1/assignment_loss‚0.7468756970544285 2loss/SRqqt2/assignment_loss‚0.6069211800519048 3loss/FBt1/assignment_loss‚0.0006340627719888895 3loss/FBt2/assignment_loss‚0.0008553805256725559 % _timestamp‚1.7845005694436696e+09 /loss/FRt1/assignment_loss‚2.209143015064395 /loss/FRt2/detection_loss‚1.5402555226484158 /loss/FBt1/detection_loss‚1.7909481786191463 /loss/FBt2/detection_loss‚0.7311830056612106 epoch‚11 _runtime‚ 3643.35432004 _step‚734 /loss/FRt2/assignment_loss‚1.112395479530096 1loss/SRqqt1/detection_loss‚1.1411754661542062 trainer/global_step‚9049 /loss/FRt1/detection_loss‚0.5863789854873004 0loss/SRqqt2/detection_loss‚0.678233862103453 &loss/total_loss‚11.144999835672218Þ;ø.cÀ‚(:Ø Û¢õÒ竲+ proc.cpu.threads‚17 network.sent‚ 1.2065765e+07% disk./.usageGB‚1173.8747444152832 disk.dm-0.in‚ 19.68359375" disk.dm-1.out‚2964.18408203125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.nvme1n1.out‚ 19167.59375 disk.md0.out‚ 107.578125 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375( disk.nvme0c0n1.out‚39630.76025390625" disk.nvme1c1n1.out‚ 19167.59375 disk.md0.in‚0& disk.nvme0n1.out‚39630.76025390625$ memory_percent‚98.82471561431885( proc.memory.availableMB‚ 192.55859375 network.recv‚ 1.082972e+06) disk./.usagePercent‚76.46147170918634" disk.dm-0.out‚55604.4130859375 disk.nvme0n1.in‚ 16.43359375# proc.memory.rssMB‚ 2875.52734375) proc.memory.percent‚17.55082607269287 cpu‚26.304427012055037”¨›Á‚(: Û¢õÒØÞ+ gpu.0.gpu‚70.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚115.339' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.446333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©÷”¢b‚(:× ê¢õÒ•æ+ network.recv‚ 1.087021e+06' disk.nvme1c1n1.out‚19168.3525390625 disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0& disk.nvme0c0n1.out‚39635.435546875$ memory_percent‚98.82621765136719) disk./.usagePercent‚76.46195697893131" disk.dm-0.out‚55609.2294921875 disk.rbd0.in‚0$ proc.memory.availableMB‚192.3125) proc.memory.percent‚17.55082607269287 network.sent‚ 1.2129893e+07% disk.nvme1n1.out‚19168.3525390625# proc.memory.rssMB‚ 2875.52734375 cpu‚26.303109671737257 proc.cpu.threads‚17$ disk./.usageGB‚1173.882194519043 disk.nvme0n1.in‚ 16.43359375$ disk.nvme0n1.out‚39635.435546875 disk.nvme1c1n1.in‚3.25 disk.md0.out‚ 107.796875 disk.dm-1.out‚2964.302734375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375‰jaˆšÃ‚(: ê¢õÒåžÕ+ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚131.741' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.91366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æFí2Ä- _wandbruntime‚3662  _runtime‚3662%è©ÅÅ¿ & lr-AdamW/pg2‚9.094133114451405e-05 trainer/global_step‚9099 % _timestamp‚1.7845005880888486e+09 _runtime‚ 3662.0013816 _step‚735 & lr-AdamW/pg1‚9.094133114451405e-05Og|)ÊÆÄ trainer/global_step‚9099 % _timestamp‚1.7845005880888486e+09 _runtime‚ 3662.0013816 _step‚735 & lr-AdamW/pg1‚9.094133114451405e-05 & lr-AdamW/pg2‚9.094133114451405e-05ßf‘ߧ2Ç- _wandbruntime‚3662  _runtime‚3662.½üÖ'ÈÐO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92245862 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40076067 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚ 3662.0014424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161483097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350742659582976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6401565204992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357144207748608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 % _timestamp‚1.7845005880894902e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346198464285584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69237237 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346198447246224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357144312809984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92256005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69237237 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69237235 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341610537261056 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92245862 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6401565204992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚352580019155009 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚350742642543616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352580122342843 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6381571908785 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69237235 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401653227008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23383410 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚350742659582976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341610327749632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6588230434816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350742642543616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6381658057259 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357144207748608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357144207748608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346198447246224àäe“Ù2Ê- _wandbruntime‚3662  _runtime‚3662ôEWÎâ&ËÜM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350780008910848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69244556 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚352617567174213 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161500316 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6382247860229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92255760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92265917 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341646722030080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350780025950208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40080357 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335057817375744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚350780008910848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402331343872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161500316 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352617670362159 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23386079 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69244558 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341646933631488 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63466170 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6588929558016 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346235319313984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9099 _runtime‚3662.254991309 _step‚737 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69244556 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6402243314688 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357182357294080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23385816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161510475 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92255760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357182357294080 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92265917 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161510475 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2©µzYç&ÌáM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346235319313984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9099 _runtime‚3662.254991309 _step‚737 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69244556 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6402243314688 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357182357294080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23385816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161510475 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92255760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357182357294080 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92265917 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161510475 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350780008910848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69244556 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚352617567174213 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161500316 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6382247860229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92255760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92265917 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341646722030080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350780025950208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40080357 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335057817375744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚350780008910848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402331343872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161500316 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352617670362159 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23386079 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69244558 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341646933631488 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63466170 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6588929558016 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes íÚŽ`%.large_pool.current‚17039360 trainer/global_step‚9149 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343430023290880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359046568913408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69603285 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6435590968832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63796879 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343430241190912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6623168113664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6435502946816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40290567 _runtime‚3680.883844046 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352610977944576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92742614 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63797129 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23506315 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348042050634224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6623361110016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6415487532667 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352610960905216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6435502946816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336806880080896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336806855177216 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359046463852032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92752757 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352610977944576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6435590968832 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348042067673584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92742614 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92752757 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352610960905216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162345899 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6415401384193 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69603287 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40290564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚359046568913408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359046463852032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92752757 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352610977944576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6435590968832 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348042067673584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92742614 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92752757 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352610960905216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162345899 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6415401384193 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69603287 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40290564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚359046568913408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚9149 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343430023290880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359046568913408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69603285 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6435590968832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63796879 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343430241190912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6623168113664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6435502946816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40290567 _runtime‚3680.883844046 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352610977944576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92742614 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816ä°Üÿ2Ø- _wandbruntime‚3681  _runtime‚3681=ŒÈâ&ÙÜM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6436181056512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63803943 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚3681.138028318 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69610608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354495103225567 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162363118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162373277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6624051880448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348078939741344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352648327272448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23509086 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23508840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336842550334464 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6416077335637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336842575238144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚359084508328960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92762669 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845006072269115e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69610606 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92752512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6436181056512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6436269085696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352648327272448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359084613397504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _step‚741 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63803694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359084613397504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343466413420032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92762669 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6416077335637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336842575238144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚359084508328960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92762669 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845006072269115e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69610606 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92752512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6436181056512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 64362¢µX<[69085696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352648327272448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359084613397504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _step‚741 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63803694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359084613397504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343466413420032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92762669 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69610608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354495103225567 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162363118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162373277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6624051880448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348078939741344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352648327272448åÄ«˜µ2Û- _wandbruntime‚3681  _runtime‚3681ï³âŽÜ€ $ _timestamp‚1.784500607227837e+09 0loss/FRt1/assignment_loss‚2.2216618338409737 0loss/FBt1/assignment_loss‚0.0712627676352523 /loss/FRt2/detection_loss‚1.4760659377588086 epoch‚11 trainer/global_step‚9149 /loss/FBt2/detection_loss‚0.5653980188201078 2loss/SRqqt1/assignment_loss‚0.6586941497385381 2loss/SRqqt2/assignment_loss‚0.6279376751732831 /loss/FRt1/detection_loss‚0.6116611093631203 _runtime‚3681.138542809 $loss/total_loss‚9.99128911276544 _step‚742 0loss/FRt2/assignment_loss‚1.0261159601460499 2loss/FBt2/assignment_loss‚0.013164026989456986 1loss/SRqqt1/detection_loss‚1.2444932224499363 1loss/SRqqt2/detection_loss‚0.7554078997752226 /loss/FBt1/detection_loss‚0.7194265110746902›†¥ˆ Ý… 0loss/FRt2/assignment_loss‚1.0261159601460499 2loss/FBt2/assignment_loss‚0.013164026989456986 1loss/SRqqt1/detection_loss‚1.2444932224499363 1loss/SRqqt2/detection_loss‚0.7554078997752226 /loss/FBt1/detection_loss‚0.7194265110746902 $ _timestamp‚1.784500607227837e+09 0loss/FRt1/assignment_loss‚2.2216618338409737 0loss/FBt1/assignment_loss‚0.0712627676352523 /loss/FRt2/detection_loss‚1.4760659377588086 epoch‚11 trainer/global_step‚9149 /loss/FBt2/detection_loss‚0.5653980188201078 2loss/SRqqt1/assignment_loss‚0.6586941497385381 2loss/SRqqt2/assignment_loss‚0.6279376751732831 /loss/FRt1/detection_loss‚0.6116611093631203 _runtime‚3681.138542809 $loss/total_loss‚9.99128911276544 _step‚742æxÊgÞ‚(:Ü ˆ£õÒˆåø+ disk.nvme1c1n1.in‚3.25' proc.memory.availableMB‚ 192.4609375) proc.memory.percent‚17.55082607269287 cpu‚26.299976906804883 proc.cpu.threads‚17 disk.dm-1.out‚ 2964.53515625 disk.rbd0.in‚0 network.sent‚ 1.2194157e+07$ disk./.usageGB‚1173.899700164795 disk.md0.out‚ 108.40234375 disk.dm-0.in‚ 19.68359375# disk.dm-0.out‚58570.33740234375$ disk.nvme1n1.out‚19177.318359375& disk.nvme0n1.out‚42589.11279296875& disk.nvme1c1n1.out‚19177.318359375# memory_percent‚98.8253116607666# proc.memory.rssMB‚ 2875.52734375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375( disk.nvme0c0n1.out‚42589.11279296875 network.recv‚ 1.093452e+06( disk./.usagePercent‚76.4630972261713 disk.md0.in‚0 disk.nvme0n1.in‚ 16.43359375 disk.nvme1n1.in‚3.25 disk.dm-1.in‚0¹;Ú,ß‚(:„ ˆ£õÒ€²Å, gpu.0.gpu‚16.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.282' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07?–92à- _wandbruntime‚3699  _runtime‚3699Ë“‰Äá¾ _runtime‚3699.847543266 _step‚743 % lr-AdamW/pg1‚9.06890137635829e-05 % lr-AdamW/pg2‚9.06890137635829e-05 trainer/global_step‚9199 $ _timestamp‚1.784500625932803e+09R+vÉâà % lr-AdamW/pg1‚9.06890137635829e-05 % lr-AdamW/pg2‚9.06890137635829e-05 trainer/global_step‚9199 $ _timestamp‚1.784500625932803e+09 _runtime‚3699.847543266 _step‚743çCHs2ã- _wandbruntime‚3699  _runtime‚3699¶CÆS×'äÑO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354479279266816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚354479296306176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6469440688640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23629486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69969335 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚9199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64134553 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6658121466368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360948719955456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40505067 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69969335 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360948825016832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69969337 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360948719955456 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356334884881825 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6469528710656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚354479296306176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6469528710656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93239366 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354479279266816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338591613039616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356334988069659 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345249941920256 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784500625937286e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163218846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6469528710656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93239366 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354479279266816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338591613039616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356334988069659 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345249941920256 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784500625937286e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163218846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360948719955456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40505067 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69969335 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360948825016832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69969337 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360948719955456 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0èÙ»ÄP2æ- _wandbruntime‚3700  _runtime‚3700³,×4çÜM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6470118798336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354516645634048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69976656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338627308196864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6470118798336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345286339484672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23631985 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345286121591808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338627333100544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356372432901029 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚745 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93249264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69976658 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349922526089984 trainer/global_step‚9199 % _timestamp‚1.7845006261921558e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40509357 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360986869500928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6449906811045 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚354516662673408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6658813394944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163225920 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360986869500928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6449992959631 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40509354 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163236079 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6470206827520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚354516645634048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23631729 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349922543129344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659006384128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3700.103234903 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64141083 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69976658 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93259421 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163236079 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69976656 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93259421 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93249264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256±iç&èáM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163236079 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6470206827520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚354516645634048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23631729 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349922543129344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659006384128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3700.103234903 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64141083 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69976658 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93259421 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163236079 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69976656 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93259421 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93249264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6470118798336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354516645634048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69976656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338627308196864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6470118798336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345286339484672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23631985 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345286121591808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338627333100544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356372432901029 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚745 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93249264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69976658 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349922526089984 trainer/global_step‚9199 % _timestamp‚1.7845006261921558e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40509357 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360986869500928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6449906811045 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚354516662673408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6658813394944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163225920 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360986869500928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6449992959631 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40509354 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362850976058880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93746261 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351729274449584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784500645792239e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23751266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6693066514432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351729257410224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6483146483483 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚748 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164081648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64471087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64470830 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340376370902016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93746261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6503466452480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340376395805696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚3719.703155099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356347597628416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358212420933067 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70335385 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6503378430464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40719564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356347614667776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6503378430464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6503466452480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40719567 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347069646927872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358212317745233 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70335387 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356347597628416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347069437416448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6693251122176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362850976058880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93746261 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351729274449584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784500645792239e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23751266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6693066514432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351729257410224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6483146483483 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚748 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164081648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64471087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64470830 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340376370902016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93746261 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6503466452480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340376395805696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚3719.703155099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356347597628416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358212420933067 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70335385 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6503378430464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40719564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356347614667776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6503378430464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6503466452480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40719567 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347069646927872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358212317745233 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70335387 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356347597628416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347069437416448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6693251122176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6483822435039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6693766751744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93746016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23753781 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚356384963995648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70342706 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93756173 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164088722 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164098881 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356384963995648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164098881 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362889125604352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6504056540160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70342708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40723857 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚3719.978867739 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚356384981035008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6693953449472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358249865764437 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚749 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64477635 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351766146517344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347106044412416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358249968952383 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362889020535808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347105832811008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360§Ó²Aæ&öàM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64477635 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351766146517344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347106044412416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358249968952383 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362889020535808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347105832811008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6504144569344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6483822435039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6693766751744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93746016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23753781 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚356384963995648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70342706 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93756173 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164088722 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164098881 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356384963995648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164098881 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362889125604352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6504056540160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70342708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40723857 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚3719.978867739 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚356384981035008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6693953449472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358249865764437 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚749í8¯‚@2÷- _wandbruntime‚3719  _runtime‚3719ȼEø‚ 2loss/SRqqt1/assignment_loss‚0.4956510690378089 2loss/FBt1/assignment_loss‚0.002161105130330654 2loss/FBt2/assignment_loss‚0.000893204201109292 0loss/SRqqt1/detection_loss‚1.090574966486672 1loss/SRqqt2/detection_loss‚0.7038727029918157 $loss/total_loss‚9.33092381635513 trainer/global_step‚9249 2loss/SRqqt2/assignment_loss‚0.6482405655872726 /loss/FBt2/detection_loss‚0.5413143219261923 % _timestamp‚1.7845006460691843e+09 _step‚750 0loss/FRt2/assignment_loss‚0.9766895377889593 /loss/FBt1/detection_loss‚0.6730651784688234 0loss/FRt1/assignment_loss‚2.1076330127102314 /loss/FRt1/detection_loss‚0.5393513473080146 /loss/FRt2/detection_loss‚1.5514768047178997 _runtime‚3719.979965411 epoch‚11vX¯ ù‡ 2loss/FBt2/assignment_loss‚0.000893204201109292 0loss/SRqqt1/detection_loss‚1.090574966486672 1loss/SRqqt2/detection_loss‚0.7038727029918157 $loss/total_loss‚9.33092381635513 trainer/global_step‚9249 2loss/SRqqt2/assignment_loss‚0.6482405655872726 /loss/FBt2/detection_loss‚0.5413143219261923 % _timestamp‚1.7845006460691843e+09 _step‚750 0loss/FRt2/assignment_loss‚0.9766895377889593 /loss/FBt1/detection_loss‚0.6730651784688234 0loss/FRt1/assignment_loss‚2.1076330127102314 /loss/FRt1/detection_loss‚0.5393513473080146 /loss/FRt2/detection_loss‚1.5514768047178997 _runtime‚3719.979965411 epoch‚11 2loss/SRqqt1/assignment_loss‚0.4956510690378089 2loss/FBt1/assignment_loss‚0.002161105130330654îñä|Åjú‚(:ß ¦£õÒ¹­©+) disk./.usagePercent‚76.46161557563914 disk.dm-0.in‚ 19.68359375" disk.dm-1.out‚2964.74658203125( disk.nvme1c1n1.out‚19182.94189453125) proc.memory.percent‚17.55082607269287 disk.md0.out‚ 109.08984375" disk.dm-0.out‚59186.0732421875 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375% disk.nvme0n1.out‚43200.9248046875 disk.nvme1n1.in‚3.25 network.sent‚ 1.2319401e+07 network.recv‚ 1.10205e+06! disk./.usageGB‚1173.876953125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚43200.9248046875 disk.nvme1c1n1.in‚3.25& disk.nvme1n1.out‚19182.94189453125# proc.memory.rssMB‚ 2875.52734375 disk.md0.in‚0 disk.nvme0n1.in‚ 16.43359375$ memory_percent‚98.82876873016357( proc.memory.availableMB‚ 191.89453125 cpu‚26.302196153302983 proc.cpu.threads‚17çò²¿›û‚(: ¦£õÒªÖƒ, gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚198.406' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.13533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(‚ÿv©ü‚(:ä µ£õÒþéÄ+ disk.dm-1.in‚0( proc.memory.availableMB‚ 192.06640625# proc.memory.rssMB‚ 2875.52734375 network.recv‚ 1.106555e+06! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375% disk.nvme0n1.out‚43208.3974609375( disk.nvme1c1n1.out‚19184.61181640625$ memory_percent‚98.82771968841553 proc.cpu.threads‚17 network.sent‚ 1.2383179e+07% disk./.usageGB‚1171.9029998779297 disk.rbd0.out‚0& disk.nvme1n1.out‚19184.61181640625 disk.md0.in‚0 disk.dm-0.in‚ 19.68359375" disk.dm-0.out‚59194.0595703125" disk.dm-1.out‚2964.87548828125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚43208.3974609375ª'³ëÆ disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.md0.out‚ 109.57421875) proc.memory.percent‚17.55082607269287 cpu‚26.300583371653776) disk./.usagePercent‚76.33304021351114Æ‘ ›ý‚(: µ£õÒõùô+ gpu.0.gpu‚32.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.683' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.894333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʎϸ2þ- _wandbruntime‚3739  _runtime‚3739·e1˜ÇÿÁ & lr-AdamW/pg1‚9.043414582184372e-05 & lr-AdamW/pg2‚9.043414582184372e-05 trainer/global_step‚9299 % _timestamp‚1.7845006657916095e+09 _runtime‚3739.703623076 _step‚751RÍf|Ì€Æ % _timestamp‚1.7845006657916095e+09 _runtime‚3739.703623076 _step‚751 & lr-AdamW/pg1‚9.043414582184372e-05 & lr-AdamW/pg2‚9.043414582184372e-05 trainer/global_step‚9299ïºM82- _wandbruntime‚3739  _runtime‚3739 ôBRØ'‚ÒO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64808577 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358215915990016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70701435 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70701437 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70701435 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94232870 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40934064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342161128764416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353572860798224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚364753232162304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70701437 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94232870 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6537404194304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6727973518848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6537404194304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3739.704416078 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40934067 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6516975958891 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164944450 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348889102283264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23874266 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358215915990016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353572877837584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358215933029376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342161153668096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364753232162304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94243013 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089750608641 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358215933029376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6728162320896:CÔÝ'ƒ×O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70701435 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70701437 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70701435 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94232870 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40934064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342161128764416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353572860798224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚364753232162304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70701437 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94232870 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6537404194304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6727973518848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6537404194304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3739.704416078 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40934067 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6516975958891 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164944450 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348889102283264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23874266 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358215915990016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353572877837584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358215933029376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342161153668096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364753232162304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94243013 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089750608641 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358215933029376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6728162320896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64808577 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358215915990016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ð&—æç2„- _wandbruntime‚3739  _runtime‚3739i Æâ&…ÜM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358253282357248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40938357 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70708758 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6517565761861 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94252925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6728671280128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94252925 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94242768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164961683 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358253282357248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164961683 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70708756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚9299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70708756 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6538082311168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6728857977856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358253299396608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364791381707776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6517651910447 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64815314 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6538082311168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164951524 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64815062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40938354 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353609732865984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342196848825344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342196823921664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364791381707776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚753†f¤’†áM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6517651910447 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64815314 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6538082311168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164951524 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64815062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40938354 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚353609732865984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342196848825344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342196823921664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364791381707776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚753 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358253282357248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40938357 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70708758 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6517565761861 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94252925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6728671280128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94252925 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94242768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164961683 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358253282357248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164961683 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70708756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚9299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70708756 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6538082311168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6728857977856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358253299396608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364791381707776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238ñRà­2‡- _wandbruntime‚3739  _runtime‚3739;ˆ 1loss/SRqqt1/detection_loss‚1.0026562655829403 %loss/total_loss‚9.882065171583578 epoch‚11 0loss/FRt1/assignment_loss‚1.9683649178192233 1loss/SRqqt1/assignment_loss‚0.724468847694456 2loss/SRqqt2/assignment_loss‚0.6020103016891855 2loss/FBt2/assignment_loss‚0.023556113908563988 /loss/FRt1/detection_loss‚0.5629114487224934 trainer/global_step‚9299 $ _timestamp‚1.784500666065996e+09 _runtime‚3739.976688087 _step‚754 1loss/FBt1/assignment_loss‚0.04245733355588715 1loss/SRqqt2/detection_loss‚0.6718230577671658 /loss/FBt1/detection_loss‚0.7829627788418697 /loss/FBt2/detection_loss‚0.8880550423839494 0loss/FRt2/assignment_loss‚1.0752862321753656 /loss/FRt2/detection_loss‚1.5375128314424786:^‘Ú ‰† /loss/FBt1/detection_loss‚0.7829627788418697 /loss/FBt2/detection_loss‚0.8880550423839494 0loss/FRt2/assignment_loss‚1.0752862321753656 /loss/FRt2/detection_loss‚1.5375128314424786 1loss/SRqqt1/detection_loss‚1.0026562655829403 %loss/total_loss‚9.882065171583578 epoch‚11 0loss/FRt1/assignment_loss‚1.9683649178192233 1loss/SRqqt1/assignment_loss‚0.724468847694456 2loss/SRqqt2/assignment_loss‚0.6020103016891855 2loss/FBt2/assignment_loss‚0.023556113908563988 /loss/FRt1/detection_loss‚0.5629114487224934 trainer/global_step‚9299 $ _timestamp‚1.784500666065996e+09 _runtime‚3739.976688087 _step‚754 1loss/FBt1/assignment_loss‚0.04245733355588715 1loss/SRqqt2/detection_loss‚0.6718230577671658òòôTŠ‚(:É Ä£õÒˆôý+" disk.dm-1.out‚2965.02197265625# proc.memory.rssMB‚ 2875.52734375) proc.memory.percent‚17.55082607269287 cpu‚26.298913650792414 network.sent‚ 1.2398944e+07 disk.nvme0n1.in‚ 16.43359375 memory_percent‚ 98.828125 proc.memory.availableMB‚192 proc.cpu.threads‚17 network.recv‚ 1.109477e+06) disk./.usagePercent‚76.46843941742425% disk./.usageGB‚1173.9817161560059! disk.nvme0n1.out‚ 43213.828125 disk.nvme1c1n1.in‚3.25& disk.nvme1n1.out‚20957.10498046875 disk.md0.out‚ 110.015625" disk.dm-0.out‚60970.8779296875 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375# disk.nvme0c0n1.out‚ 43213.828125( disk.nvme1c1n1.out‚20957.10498046875 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0 disk.md0.in‚01䂸‹‚(:„ Ä£õÒõÓ¤, gpu.0.gpu‚51.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚197.661' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.887 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xXfæ2Œ- _wandbruntime‚3759  _runtime‚3759œn;Å¿ % _timestamp‚1.7845006860748928e+09 _runtime‚3759.985834993 _step‚755 % lr-AdamW/pg1‚9.03057604870803e-05 % lr-AdamW/pg2‚9.03057604870803e-05 trainer/global_step‚9349¨x WÊŽÄ _runtime‚3759.985834993 _step‚755 % lr-AdamW/pg1‚9.03057604870803e-05 % lr-AdamW/pg2‚9.03057604870803e-05 trainer/global_step‚9349 % _timestamp‚1.7845006860748928e+09ó /ÿ¬2- _wandbruntime‚3759  _runtime‚3759iöˆ¼Ø'ÒO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366655488265728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343945886626816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360084251390976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6571341936128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360084234351616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71067485 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6550805434299 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚9349 _runtime‚3759.986830135 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94739765 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366655488265728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355416481225584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6571253914112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65146299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360084234351616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚355416464186224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350709040780800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350708837560832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343945911530496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366655593327104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6571341936128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165797107 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23997732 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165807252 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71067487 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94739765 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41148567 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚361967286659883 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6550719285825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41148564 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71067487 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94729622 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71067485 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360084251390976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0^ÆÓÝ'‘×O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚9349 _runtime‚3759.986830135 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94739765 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366655488265728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355416481225584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6571253914112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65146299 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360084234351616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚355416464186224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350709040780800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350708837560832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343945911530496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366655593327104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6571341936128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165797107 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23997732 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165807252 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71067487 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94739765 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41148567 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚361967286659883 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6550719285825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41148564 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71067487 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94729622 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71067485 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360084251390976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366655488265728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343945886626816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360084251390976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6571341936128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360084234351616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71067485 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6550805434299ôƒƒ_›2’- _wandbruntime‚3760  _runtime‚3760Å\JG“ÛM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.abx½ š&llocated‚71074808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24000149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355453336253984 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360121600718848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366693637811200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 % _timestamp‚1.7845006863495572e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71074808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6763653364736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360121617758208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6763831673856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6551395237269 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366693637811200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71074806 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚ 3760.26071556 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165824485 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6571932023808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94739520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360121600718848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71074806 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94739520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350745235148800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362004834679199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6551481385855 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362004731491253 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6572020052992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41152857 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚9349 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165814326 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65152758 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23999904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350745438361600 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355453353293344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65153006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41152854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360121617758208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚757 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165814326 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6572020052992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165824485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374Z«1æ&”àM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94749677 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6571932023808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94739520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360121600718848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71074806 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94739520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350745235148800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362004834679199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6551481385855 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362004731491253 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6572020052992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41152857 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚9349 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165814326 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65152758 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23999904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350745438361600 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355453353293344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65153006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41152854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360121617758208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚757 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165814326 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6572020052992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165824485 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71074808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24000149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355453336253984 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360121600718848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366693637811200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 % _timestamp‚1.7845006863495572e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71074808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6763653364736 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360121617758208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6763831673856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6551395237269 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366693637811200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71074806 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚ 3760.26071556 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165824485 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41363067 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 " _timestamp‚1.7845007063112e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6605191655936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6584634909707 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71433537 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41363064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361952569752576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24121117 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6584548761233 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368557849430528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6797934354432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368557744369152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361952552713216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363844719523291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361952552713216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65484181 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361952569752576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95226374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6605279677952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0#…{Ú'¡ÔO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41363067 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 " _timestamp‚1.7845007063112e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6605191655936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6584634909707 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71433537 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41363064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚361952569752576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24121117 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6584548761233 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368557849430528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6797934354432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368557744369152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361952552713216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363844719523291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361952552713216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65484181 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361952569752576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95226374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6605279677952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368557744369152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚760ø{À¢¤2¢- _wandbruntime‚3780  _runtime‚3780V‘-â&£ÜM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6585224712677 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6605869765632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845007065829706e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363882164354661 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166677128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95236272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95246429 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361989919080448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368595893914624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166687287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚9399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352565179286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95236272 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65490922 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71440856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24123814 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71440856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357296939641984 _runtime‚3780.494124759 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361989936119808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6605869765632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166677128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368595788846080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363882267542607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65491171 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _step‚761 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71440858 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6605957794816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95246429 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6798636426752 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6605957794816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0‹}oêç&¤áM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6605957794816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95246429 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6798636426752 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6605957794816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6585224712677 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6605869765632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845007065829706e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363882164354661 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166677128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95236272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95246429 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361989919080448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368595893914624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166687287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚9399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352565179286016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95236272 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65490922 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71440856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24123814 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71440856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357296939641984 _runtime‚3780.494124759 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361989936119808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6605869765632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166677128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368595788846080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363882267542607 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65491171 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _step‚761 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71440858 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238ù€Ⱦ2¥- _wandbruntime‚3780  _runtime‚3780¿¾Â÷˦„ /loss/FRt1/assignment_loss‚2.066865574406541 0loss/FRt2/assignment_loss‚0.9969924980021537 0loss/SRqqt1/detection_loss‚1.130488639542212 epoch‚11 $ _timestamp‚1.784500706583977e+09 2loss/SRqqt1/assignment_loss‚0.6516664369275011 4loss/FBt2/assignment_loss‚0.00027976437290730714 /loss/FRt2/detection_loss‚1.4887853122501244 trainer/global_step‚9399 3loss/FBt1/assignment_loss‚0.0032677325231702525 1loss/SRqqt~—9þ?2/detection_loss‚0.6109270620028339 %loss/total_loss‚9.843970330762264 _runtime‚3780.494693249 2loss/SRqqt2/assignment_loss‚0.6480920971460311 /loss/FRt1/detection_loss‚0.5913528306896731 /loss/FBt1/detection_loss‚1.2364862538874148 /loss/FBt2/detection_loss‚0.4187661290117022 _step‚762ãÅ.!§‰ _step‚762 /loss/FRt1/assignment_loss‚2.066865574406541 0loss/FRt2/assignment_loss‚0.9969924980021537 0loss/SRqqt1/detection_loss‚1.130488639542212 epoch‚11 $ _timestamp‚1.784500706583977e+09 2loss/SRqqt1/assignment_loss‚0.6516664369275011 4loss/FBt2/assignment_loss‚0.00027976437290730714 /loss/FRt2/detection_loss‚1.4887853122501244 trainer/global_step‚9399 3loss/FBt1/assignment_loss‚0.0032677325231702525 1loss/SRqqt2/detection_loss‚0.6109270620028339 %loss/total_loss‚9.843970330762264 _runtime‚3780.494693249 2loss/SRqqt2/assignment_loss‚0.6480920971460311 /loss/FRt1/detection_loss‚0.5913528306896731 /loss/FBt1/detection_loss‚1.2364862538874148 /loss/FBt2/detection_loss‚0.4187661290117022ú›åEi¨‚(:Þ ñ£õÒú©®+ disk.nvme0n1.in‚ 16.43359375( disk.nvme1c1n1.out‚22734.40869140625& disk.nvme1n1.out‚22734.40869140625# proc.memory.availableMB‚191.875 proc.cpu.threads‚17) disk./.usagePercent‚76.44623453736286% disk./.usageGB‚1173.6408157348633 disk.rbd0.out‚0% disk.nvme0n1.out‚43249.6572265625 disk.nvme1n1.in‚3.25 disk.md0.out‚ 111.45703125$ memory_percent‚98.82888793945312) proc.memory.percent‚17.55082607269287 network.sent‚ 1.2516626e+07 network.recv‚ 1.12145e+06 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375' disk.nvme0c0n1.out‚43249.6572265625 disk.md0.in‚0# proc.memory.rssMB‚ 2875.52734375 cpu‚26.294340037059573 disk.dm-0.in‚ 19.68359375" disk.dm-0.out‚62780.5947265625 disk.dm-1.in‚0" disk.dm-1.out‚2965.37744140625 disk.nvme1c1n1.in‚3.25”EÇ6©‚(:„ ñ£õÒ ü…, gpu.0.gpu‚32.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.697' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.899 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fq¨?2ª- _wandbruntime‚3800  _runtime‚3800}$ïkÅ«¿ _step‚763 % lr-AdamW/pg1‚9.00470973714915e-05 % lr-AdamW/pg2‚9.00470973714915e-05 trainer/global_step‚9449 % _timestamp‚1.7845007265050871e+09 _runtime‚3800.416057745Ú§Ê¬Ä % _timestamp‚1.7845007265050871e+09 _runtime‚3800.416057745 _step‚763 % lr-AdamW/pg1‚9.00470973714915e-05 % lr-AdamW/pg2‚9.00470973714915e-05 trainer/global_step‚9449û°Â%2­- _wandbruntime‚3800  _runtime‚3800ݹ‡×'®ÑO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363820888114176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95733269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24243883 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365722049198865 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65821688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370460000472576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71799585 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24244121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365722152386699 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65821447 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639129397760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347515427255296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41577567 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6639217419776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6618378236641 trainer/global_step‚9449 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354348211520512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 $ _timestamp‚1.784500726506177e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370460105533952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370460000472576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71799587 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71799587 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347515402351616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6618464385115 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71799585 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41577564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95723126 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167522711 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639129397760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6832809168896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363820871074816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚3800.418210079 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363820888114176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354348418934784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6832991679488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚764 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370460105533952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363820871074816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167532856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359103670962224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370460000472576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71799585 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24244121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365722152386699 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65821447 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639129397760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347515427255296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41577567 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6639217419776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6618378236641 trainer/global_step‚9449 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354348211520512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 $ _timestamp‚1.784500726506177e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370460105533952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370460000472576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71799587 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71799587 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347515402351616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6618464385115 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71799585 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41577564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95723126 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167522711 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639129397760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6832809168896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363820871074816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚3800.418210079üb˜1Ã2°- _wandbruntime‚3800  _runtime‚3800hŽ$±ÛM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚9449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743181 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363858237442048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71806906 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚3800.690493973 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95733024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41581854 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347551097508864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95743181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363858254481408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65828230 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363858254481408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6833686967808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354384809380352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71806908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359140543029984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370498150018048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 $ _timestamp‚1.784500726779357e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6639895536640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚347551122412544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71806908 L>DeviceStatsMonitor.on_train_batch_end/acti/Sve_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41581857 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354384601973248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167550089 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0'ýRæ&²àM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354384809380352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71806908 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359140543029984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370498150018048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 $ _timestamp‚1.784500726779357e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6639895536640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚347551122412544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71806908 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41581857 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354384601973248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167550089 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚9449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743181 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363858237442048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71806906 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚3800.690493973 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95733024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41581854 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347551097508864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95743181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363858254481408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65828230 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363858254481408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6833686967808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152ý降2³- _wandbruntime‚3800  _runtime‚3800ö ´… /loss/FRt2/detection_loss‚1.4134013970546864 trainer/global_step‚9449 0loss/FRt2/assignment_loss‚0.9198219696680705 2loss/SRqqt1/assignment_loss‚0.5285952930470618 2loss/FBt1/assignment_loss‚0.008203129623738774 3loss/FBt2/assignment_loss‚0.0017350126609511315 /loss/FBt1/detection_loss‚0.4507733534661714 _runtime‚3800.691209144 2loss/SRqqt2/assignment_loss‚0.5731771966115524 1loss/SRqqt1/detection_loss‚1.0917623581302662 1loss/SRqqt2/detection_loss‚0.6283900834144637 %loss/total_loss‚8.747232428177847 % _timestamp‚1.7845007267804937e+09 0loss/FRt1/assignment_loss‚2.1308619267584983 /loss/FRt1/detection_loss‚0.4953373886420243 /loss/FBt2/detection_loss‚0.5051733191003618 epoch‚11 _step‚766”ÔtɵŠ 0loss/FRt1/assignment_loss‚2.1308619267584983 /loss/FRt1/detection_loss‚0.4953373886420243 /loss/FBt2/detection_loss‚0.5051733191003618 epoch‚11 _step‚766 /loss/FRt2/detection_loss‚1.4134013970546864 trainer/global_step‚9449 0loss/FRt2/assignment_loss‚0.9198219696680705 2loss/SRqqt1/assignment_loss‚0.5285952930470618 2loss/FBt1/assignment_loss‚0.008203129623738774 3loss/FBt2/assignment_loss‚0.0017350126609511315 /loss/FBt1/detection_loss‚0.4507733534661714 _runtime‚3800.691209144 2loss/SRqqt2/assignment_loss‚0.5731771966115524 1loss/SRqqt1/detection_loss‚1.0917623581302662 1loss/SRqqt2/detection_loss‚0.6283900834144637 %loss/total_loss‚8.747232428177847 % _timestamp‚1.7845007267804937e+09þ„'lPj¶‚(:ß €¤õÒϯË+$ memory_percent‚99.22454357147217# proc.memory.rssMB‚ 2875.52734375) proc.memory.percent‚17.55082607269287 cpu‚26.292724148695246! disk.nvme0c0n1.in‚ 16.43359375 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.dm-1.out‚2965.4892578125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚43345.26708984375& disk.nvme0n1.out‚43345.26708984375 disk.nvme1c1n1.in‚3.25& disk.nvme1n1.out‚22767.38525390625( proc.memory.availableMB‚ 127.05078125 network.recv‚ 1.125288e+06) disk./.usagePercent‚76.23532905075453$ disk./.usageGB‚1170.402889251709 disk.dm-0.in‚ 19.68359375 disk.dm-0.out‚62908.08984375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.43359375 disk.nvme1n1.in‚3.25 network.sent‚ 1.259184e+07( disk.nvme1c1n1.out‚22767.38525390625 disk.md0.in‚0 disk.md0.out‚ 111.921875Ág©·‚(:… €¤õÒÌ÷õ+ gpu.0.gpu‚85.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚198.483' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.161 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ–$´2¸- _wandbruntime‚3820  _runtime‚3820ÕÎæ ƹÀ & lr-AdamW/pg1‚8.991682373368095e-05 & lr-AdamW/pg2‚8.991682373368095e-05 trainer/global_step‚9499 % _timestamp‚1.7845007464425642e+09 _runtime‚ 3820.35347704 _step‚767†;ü˺Š_step‚767 & lr-AdamW/pg1‚8.991682373368095e-05 & lr-AdamW/pg2‚8.991682373368095e-05 trainer/global_step‚9499 % _timestamp‚1.7845007464425642e+09 _runtime‚ 3820.35347704ÿ÷á½þ2»- _wandbruntime‚3820  _runtime‚3820Ýš÷Ø'¼ÒO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96219878 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚360947291389584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356167916230144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6867942721024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72165635 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367599482062273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72165635 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6673155161600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365689206475776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6652207712049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96219878 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365689189436416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6673067139584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6673155161600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24367188 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356168127838720 _runtime‚3820.354446511 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168395658 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72165637 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372362256576000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72165637 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66159255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6652293860523 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365689206475776 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362256576000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41792067 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365689189436416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 _step‚768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚9499]—„X!½×O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72165637 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372362256576000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72165637 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66159255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6652293860523 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365689206475776 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362256576000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41792067 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365689189436416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 _step‚768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚9499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96219878 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚360947291389584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356167916230144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6867942721024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72165635 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367599482062273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72165635 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6673155161600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365689206475776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6652207712049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.eƒö"…on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96219878 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365689189436416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6673067139584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6673155161600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24367188 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356168127838720 _runtime‚3820.354446511 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168395658€kx!2¾- _wandbruntime‚3820  _runtime‚3820üJŽá&¿ÛM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168412891 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365726555803648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6673833278464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6673745249280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24369283 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367637133269423 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367637030081477 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72172958 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41796354 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372400301052928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66165637 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96229776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356204519258112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6868458576896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96229776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360984146417984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72172956 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6673833278464 $ _timestamp‚1.784500746712863e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96239933 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365726555803648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349335880274944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349335855371264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3820.623960564 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365726572843008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96239933 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356204313948160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168412891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360984163457344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6652969812079 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6652883663493 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41796357 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365726555803648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349335880274944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349335855371264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3820.623960564 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365726572843008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96239933 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356204313948160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168412891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360984163457344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6652969812079 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6652883663493 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41796357 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24369283 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367637133269423 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367637030081477 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72172958 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41796354 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372400301052928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66165637 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96229776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356204519258112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ÍFç2Á- _wandbruntime‚3820  _runtime‚3820y 1ˆ ƒ 1loss/FBt2/assignment_loss‚0.11015704349681661 /loss/FRt1/detection_loss‚0.5889452416093529 %loss/total_loss‚9.349305066640099 _runtime‚3820.624675815 0loss/FRt1/assignment_loss‚2.1460952010096572 2loss/SRqqt1/assignment_loss‚0.7039011738754247 2loss/SRqqt2/assignment_loss‚0.5152306763090349 2loss/FBt1/assignment_loss‚0.057878451931543395 epoch‚11 0loss/FRt2/assignment_loss‚1.0056897151703927 /loss/FRt2/detection_loss‚1.3721412768770083 1loss/SRqqt2/detection_loss‚0.5863633707791492 /loss/FBt1/detection_loss‚0.7658178796215603 trainer/global_step‚9499 % _timestamp‚1.7845007467139633e+09 _step‚770 0loss/SRqqt1/detection_loss‚1.116886477980026 0loss/FBt2/detection_loss‚0.38019855798013263u´sïÈ 0loss/FRt1/assignment_loss‚2.1460952010096572 2loss/SRqqt1/assignment_loss‚0.7039011738754247 2loss/SRqqt2/assignment_loss‚0.5152306763090349 2loss/FBt1/assignment_loss‚0.057878451931543395 epoch‚11 0loss/FRt2/assignment_loss‚1.0056897151703927 /loss/FRt2/detection_loss‚1.3721412768770083 1loss/SRqqt2/detection_loss‚0.5863633707791492 /loss/FBt1/detection_loss‚0.7658178796215603 trainer/global_step‚9499 % _timestamp‚1.7845007467139633e+09 _step‚770 0loss/SRqqt1/detection_loss‚1.116886477980026 0loss/FBt2/detection_loss‚0.38019855798013263 1loss/FBt2/assignment_loss‚0.11015704349681661 /loss/FRt1/detection_loss‚0.5889452416093529 %loss/total_loss‚9.349305066640099 _runtime‚3820.624675815‚Û#«VdÄ‚(:Ù ¤õÒú¸Î+ disk.nvme0n1.in‚ 16.43359375$ disk.nvme0n1.out‚43443.646484375 disk.dm-0.in‚ 19.68359375 disk.nvme1c1n1.in‚3.25' disk.nvme1c1n1.out‚22775.1552734375 disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.md0.out‚ 112.3671875! disk.dm-1.out‚2968.7744140625 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.129938e+06! disk.nvme0c0n1.in‚ 16.43359375& disk.nvme0c0n1.out‚43443.646484375% disk.nvme1n1.out‚22775.1552734375 disk.dm-0.out‚ 63009.984375 disk.dm-1.in‚0$ memory_percent‚98.77407550811768( proc.memory.availableMB‚ 200.85546875) disk./.usagePercent‚76.21986353131425$ disk./.usageGB‚1170.165454864502# proc.memory.rssMB‚ 2781.89453125* proc.memory.percent‚16.979336738586426 cpu‚26.29118898671864 proc.cpu.threads‚17 network.sent‚ 1.2653139e+07n '†œÅ‚(:‘ ¤õÒ¼Ýý+ gpu.0.gpu‚83.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚112.889' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.629666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü–` lÆ‚(:á ž¤õÒôŒÊ+( disk.nvme1c1n1.out‚22809.83154296875 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚43447.2392578125# memory_percent‚98.7741231918335 proc.cpu.threads‚17 network.sent‚ 1.2704411e+07) disk./.usagePercent‚76.21978700131862% disk./.usageGB‚1170.1642799377441 disk.md0.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375# proc.memory.rssMB‚ 2781.89453125 network.recv‚ 1.133583e+06& disk.nvme1n1.out‚22809.83154296875 disk.dm-0.in‚ 19.68359375 disk.dm-0.out‚63047.48828125" disk.dm-1.out‚2968.99462890625 disk.nvme0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.md0.out‚ 112.62109375( proc.memory.availableMB‚ 200.84765625* proc.memory.percent‚16.979336738586426 cpu‚26.28953681390368% disk.nvme0n1.out‚43447.23925781253¦L›Ç‚(: ž¤õÒ±Žõ+ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚199.793' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.59766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02-ži2È- _wandbruntime‚3840  _runtime‚3840¬‚‰ÎÆÉÀ & lr-AdamW/pg2‚8.978592482130577e-05 trainer/global_step‚9549 % _timestamp‚1.7845007667339408e+09 _runtime‚ 3840.64486231 _step‚771 & lr-AdamW/pg1‚8.978592482130577e-05hpðAËÊÅ _runtime‚ 3840.64486231 _step‚771 & lr-AdamW/pg1‚8.978592482130577e-05 & lr-AdamW/pg2‚8.978592482130577e-05 trainer/global_step‚9549 % _timestamp‚1.7845007667339408e+09ƒFZ#2Ë- _wandbruntime‚3840  _runtime‚3840QX_+ÍÌÒO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚772 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚367557507798016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72531685 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6707092903424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357987829914112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42006564 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24490306 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96726773 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169248315 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚367557524837376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚367557507798016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allo‘ cated‚351084942980096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169258460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚369476914925681 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362790894777584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351084918076416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚9549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚367557524837376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96716630 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6707004881408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6686123335931 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6707004881408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6686037187457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3840.647021984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72531685 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374264512679424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24490544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-11Á9Ý'Í×O F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169248315 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚367557524837376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚367557507798016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351084942980096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169258460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚369476914925681 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362790894777584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351084918076416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚9549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚367557524837376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96716630 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6707004881408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6686123335931 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6707004881408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6686037187457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3840.647021984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72531685 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374264512679424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24490544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚772 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚367557507798016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72531685 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6707092903424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357987829914112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42006564 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24490306 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96726773 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330„Úìü2Î- _wandbruntime‚3840  _runtime‚3840Aë§â&ÏÜM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169265534 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374302557156352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96726528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169265534 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6903400923136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72539006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚367594891204608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24492686 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72539008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42010854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6707771020288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72539006 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66503781 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6686713138901 % _timestamp‚1.7845007670067022e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351120638137344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362827766845344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362827749805984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72539008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96726528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚367594891204608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369514566132831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351120613233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6707682991104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24492924 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3840.917806162 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42010857 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6686799287487 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96736685 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66503540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6707771020288 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169275693 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369514462944885 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358024225758208 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚773 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928}­+¾á!ÐáM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358024225758208 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚773 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169265534 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374302557156352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96726528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169265534 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6903400923136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72539006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚367594891204608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24492686 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72539008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42010854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6707771020288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72539006 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66503781 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6686713138901 % _timestamp‚1.7845007670067022e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351120638137344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362827766845344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362827749805984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72539008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96726528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚367594891204608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369514566132831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351120613233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6707682991104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24492924 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3840.917806162 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42010857 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6686799287487 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96736685 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66503540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6707771020288 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169275693 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369514462944885 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751…B³žß2Ñ- _wandbruntime‚3840  _runtime‚3840~j Òƒ _runtime‚3840.921360969 _step‚774 /loss/FRt2/detection_loss‚1.4089336469910376 /loss/FBt2/detection_loss‚0.4626884166187105 % _timestamp‚1.7845007670077298e+09 3loss/FBt2/assignment_loss‚0.0015495853737619132 1loss/SRqqt2/detection_loss‚0.6104213059067867 /loss/FRt1/assignment_loss‚2.130641425379151 0loss/FRt2/assignment_loss‚0.9761481710842678 2loss/SRqqt1/assignment_loss‚0.5664886239831871 2loss/SRqqt2/assignment_loss‚0.6188170640300229 2loss/FBt1/assignment_loss‚0.026991486579285826 /loss/FRt1/detection_loss‚0.5828420245271297 /loss/FBt1/detection_loss‚0.8373466595434226 $loss/total_loss‚9.31714170693477 1loss/SRqqt1/detection_loss‚1.0942732969180067 epoch‚11 trainer/global_step‚95492sãÖÓˆ /loss/FRt2/detection_loss‚1.4089336469910376 /loss/FBt2/detection_loss‚0.4626884166187105 % _timestamp‚1.7845007670077298e+09 3loss/FBt2/assignment_loss‚0.0015495853737619132 1loss/SRqqt2/detection_loss‚0.6104213059067867 /loss/FRt1/assignment_loss‚2.130641425379151 0loss/FRt2/assignment_loss‚0.9761481710842678 2loss/SRqqt1/assignment_loss‚0.5664886239831871 2loss/SRqqt2/assignment_loss‚0.6188170640300229 2loss/FBt1/assignment_loss‚0.026991486579285826 /loss/FRt1/detection_loss‚0.5828420245271297 /loss/FBt1/detection_loss‚0.8373466595434226 $loss/total_loss‚9.31714170693477 1loss/SRqqt1/detection_loss‚1.0942732969180067 epoch‚11 trainer/global_step‚9549 _runtime‚3840.921360969 _step‚774†âÏ,ŠhÔ‚(:Ý ­¤õÒŽûÙ+' disk.nvme0c0n1.out‚43458.7353515625 disk.nvme1n1.in‚3.25% proc.memory.availableMB‚ 200.53125# proc.memory.rssMB‚ 2781.89453125 network.recv‚ 1.137146e+06) disk./.usagePercent‚76.21981731514805% disk./.usageGB‚1170.1647453308105 disk.md0.in‚0 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 disk.md0.out‚ 113.078125 disk.dm-0.out‚63059.13671875" disk.dm-1.out‚2969.12744140625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375% disk.nvme0n1.out‚43458.7353515625& disk.nvme1n1.out‚22811.08544921875$ memory_percent‚98.77605438232422 disk.rbd0.out‚0( disk.nvme1c1n1.out‚22811.08544921875* proc.memory.percent‚16.979336738586426 cpu‚26.28795983980622 proc.cpu.threads‚17 network.sent‚ 1.2720213e+07ar6Õ‚(:„ ­¤õÒÛƒ², gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚115.857' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.619 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F2Ö- _wandbruntime‚3860  _runtime‚3860LÞ±Ç×Á % _timestamp‚1.7845007869922285e+09 _runtime‚3860.903186451 _step‚775 & lr-AdamW/pg1‚8.965440273097774e-05 & lr-AdamW/pg2‚8.965440273097774e-05 trainer/global_step‚9599À,_ÌØÆ & lr-AdamW/pg1‚8.965440273097774e-05 & lr-AdamW/pg2‚8.965440273097774e-05 trainer/global_step‚9599 % _timestamp‚1.7845007869922285e+09 _runtime‚3860.903186451 _step‚775‡£½‹¨2Ù- _wandbruntime‚3860  _runtime‚3860˜È¼EØ'ÚÒO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66834840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352869700842496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42221064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚369425826159616 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97223525 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24613536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937655119872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72897737 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369425843198976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6740942623232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376166873844224 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376166768782848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6741030645248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369425843198976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72897737 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359807534278656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937833436160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97223525 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66834600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24613773 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6719952811339 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97213382 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376166768782848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6741030645248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369425843198976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72897737 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359807534278656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937833436160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97223525 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66834600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24613773 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6719952811339 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97213382 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912ˆï/@e2Ü- _wandbruntime‚3861  _runtime‚3861&ÿ„5zÝÛM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚376204918328320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369463192526848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938353241600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚9599 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66841226 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369463192526848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97233437 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6741620732928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170138495 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170138495 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42225354 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359843933841920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72905056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369463209566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170128336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚3861.175802132 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369463209566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720628762895 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72905058 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66841501 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720542614309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97223280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6741708762112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24615872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42225357 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚371391895808293 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364671370233344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97223280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938537842176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364671353193984ñH¼‹æ&ÞàM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚376204918328320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369463192526848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938353241600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚9599 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66841226 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369463192526848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97233437 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6741620732928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170138495 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170138495 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42225354 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359843933841920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72905056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚369463209566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170128336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚3861.175802132 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369463209566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720628762895 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72905058 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66841501 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720542614309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97223280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6741708762112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24615872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42225357 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚371391895808293 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364671370233344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97223280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6938537842176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364671353193984‰›XÙ/2ß- _wandbruntime‚3861  _runtime‚3861c`à‚ 1loss/SRqqt1/detection_loss‚1.0460719012478792 /loss/FBt2/detection_loss‚0.5613095091236099 %loss/total_loss‚9.557833634891967 trainer/global_step‚9599 0loss/FRt2/assignment_loss‚0.8624612582157333 /loss/FRt2/detection_loss‚1.3985462859013225 0loss/SRqqt2/detection_loss‚0.660880036078774 /loss/FBt1/detection_loss‚1.0497291010251502 $ _timestamp‚1.784500787265738e+09 _runtime‚3861.179383649 0loss/FRt1/assignment_loss‚2.0623171029568734 2loss/SRqqt2/assignment_loss‚0.5760254179006561 2loss/FBt2/assignment_loss‚0.049338384947480135 /loss/FRt1/detection_loss‚0.5587471018318736 epoch‚11 _step‚778 2loss/SRqqt1/assignment_loss‚0.7239627604664979 2loss/FBt1/assignment_loss‚0.008444775196115492+×½ï ᇠ/loss/FBt2/detection_loss‚0.5613095091236099 %loss/total_loss‚9.557833634891967 trainer/global_step‚9599 0loss/FRt2/assignment_loss‚0.8624612582157333 /loss/FRt2/detection_loss‚1.3985462859013225 0loss/SRqqt2/detection_loss‚0.660880036078774 /loss/FBt1/detection_loss‚1.0497291010251502 $ _timestamp‚1.784500787265738e+09 _runtime‚3861.179383649 0loss/FRt1/assignment_loss‚2.0623171029568734 2loss/SRqqt2/assignment_loss‚0.5760254179006561 2loss/FBt2/assignment_loss‚0.049338384947480135 /loss/FRt1/detection_loss‚0.5587471018318736 epoch‚11 _step‚778 2loss/SRqqt1/assignment_loss‚0.7239627604664979 2loss/FBt1/assignment_loss‚0.008444775196115492 1loss/SRqqt1/detection_loss‚1.0460719012478792Šµ[«kâ‚(:à ¼¤õÒ–“Á- disk.md0.in‚0! disk.nvme0c0n1.in‚ 16.43359375# proc.memory.rssMB‚ 2833.21484375) proc.memory.percent‚17.29257106781006) disk./.usagePercent‚76.21981135177177 disk.nvme1c1n1.in‚3.25' disk.nvme1c1n1.out‚22811.9423828125 disk.dm-1.in‚0& disk.nvme0n1.out‚43459.48388671875 disk.nvme1n1.in‚3.25% disk.nvme1n1.out‚22811.9423828125 disk.md0.out‚ 113.328125 disk.dm-0.in‚ 19.68359375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.08840656280518 network.sent‚ 1.2769873e+07 disk.nvme0n1.in‚ 16.43359375 disk.dm-0.out‚63059.85546875" disk.dm-1.out‚2969.47021484375( disk.nvme0c0n1.out‚43459.48388671875( proc.memory.availableMB‚ 149.35546875 cpu‚26.289397152911626 proc.cpu.threads‚17 network.recv‚ 1.141108e+06% disk./.usageGB‚1170.1646537780762M*ž¾›ã‚(: ¼¤õ҆Ϥ, gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚181.643' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚60.547666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yúÍ2ä- _wandbruntime‚3882  _runtime‚3882a•×¹±=å«{ :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23392075300216675 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16502133011817932 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12165867537260056 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15901066362857819 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24672405421733856 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6284631490707397 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07998766005039215 ,jet/accuracy_4_of_4‚0.014682539738714695 particle/accuracy_1_of_2‚1 trainer/global_step‚9611 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6743000745773315 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.479549765586853 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6150804758071899 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1371985226869583 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010043350979685783 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6309456825256348 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7410784959793091 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13812433183193207 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9922407865524292 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17824074625968933 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7232633233070374 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14094148576259613 particle/accuracy_3_of_5‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898862361907959 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16502133011817932 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09849347174167633 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7691681385040283 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 _step‚779 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41002869606018066 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12538640201091766 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8214285373687744 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037433426827192307 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.031292520463466644 +jet/accuracy_1_of_1‚0.30309587717056274 +particle/specificity‚0.9113221764564514 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8273504376411438 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9946743249893188 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5435090661048889 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2858682870864868 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6062811613082886 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_2_of_4‚0.787034809589386 'particle/f_score‚0.6513764262199402 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16298796236515045 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7045790553092957 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39840349555015564 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13958343863487244 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15828731656074524 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_3_of_4‚0.09573765844106674 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 epoch‚11 (loss/val_total_loss‚9.09969343864925 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6059672236442566 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9059392809867859 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9821584820747375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7264662981033325 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6874260902404785 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.031292520463466644 /particle/accuracy_3_of_4‚0.9965986609458923 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17696163058280945 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047915101051331 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15341684222221375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8529885411262512 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5524417161941528 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962959289550781 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_3_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8149122595787048 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9853572845458984 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 +jet/accuracy_2_of_2‚0.14053361117839813 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1073177233338356 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6250393390655518 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004882104694843292 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03244419023394585 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_5_of_5‚0 % _timestamp‚1.7845008081924815e+09 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8034361004829407 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9643168449401855 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.622542679309845 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17824074625968933 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026051048189401627 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6250000596046448 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5576664209365845 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6301707029342651 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7388700842857361 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6678752899169922 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08819051831960678 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18428996205329895 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 jet/accuracy_1_of_5‚1 _runtime‚3882.103883852 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932602643966675 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7248427271842957 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032626572996377945 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47183653712272644 *jet/accuracy_3_of_3‚0.0733180046081543 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6150804758071899 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7327925562858582 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5325521230697632 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1808464378118515 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6689592003822327 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8554471135139465 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9924231171607971 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5884274244308472 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8575102686882019 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9900393486022949 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 +particle/sensitivity‚0.6396735310554504 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4353642761707306 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6987022161483765 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37962958216667175 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6369109749794006 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7285783290863037 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7692018151283264 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7474358677864075 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13758061826229095 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9184287190437317 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7586855888366699 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1373296082019806 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3698500096797943 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.033826105296611786 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.821947455406189 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940996766090393 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8469277620315552 particle/accuracy_2_of_3‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6236093640327454 5validation_average_jet_accuracy‚0.347744882106781 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15201427042484283 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6789673566818237 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6284631490707397 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013283821754157543 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5554324388504028 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8469277620315552 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6169036030769348 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9713048338890076 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917598962783813 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025067295879125595 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970799684524536 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_2_of_3‚0.5280469655990601 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.672034740447998 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03371036425232887 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6537906527519226 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9791263341903687 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2892066538333893 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8149122595787048 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3672877252101898 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5432364344596863 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9937539100646973 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15946100652217865 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_4_of_4‚0.9498400092124939 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 )jet/accuracy_1_of_3‚0.914411187171936 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980652332305908 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820030927658081 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49007564783096313 ("Purity/1FRt2SRqqt1FBt/event_purity‚0/B=æ°{ 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1808464378118515 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6689592003822327 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8554471135139465 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9924231171607971 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5884274244308472 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8575102686882019 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9900393486022949 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 +particle/sensitivity‚0.6396735310554504 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4353642761707306 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6987022161483765 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37962958216667175 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6369109749794006 @&Purity/0FRt2SRqqt1FBt/event_proportion‚wô{y70.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7285783290863037 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7692018151283264 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7474358677864075 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13758061826229095 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9184287190437317 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7586855888366699 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1373296082019806 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3698500096797943 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.033826105296611786 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.821947455406189 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940996766090393 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8469277620315552 particle/accuracy_2_of_3‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6236093640327454 5validation_average_jet_accuracy‚0.347744882106781 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15201427042484283 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6789673566818237 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6284631490707397 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013283821754157543 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5554324388504028 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8469277620315552 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6169036030769348 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9713048338890076 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917598962783813 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025067295879125595 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970799684524536 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_2_of_3‚0.5280469655990601 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.672034740447998 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03371036425232887 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6537906527519226 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9791263341903687 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2892066538333893 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8149122595787048 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3672877252101898 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5432364344596863 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9937539100646973 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15946100652217865 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_4_of_4‚0.9498400092124939 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 )jet/accuracy_1_of_3‚0.914411187171936 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980652332305908 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820030927658081 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49007564783096313 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23392075300216675 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16502133011817932 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12165867537260056 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15901066362857819 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24672405421733856 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6284631490707397 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07998766005039215 ,jet/accuracy_4_of_4‚0.014682539738714695 particle/accuracy_1_of_2‚1 trainer/global_step‚9611 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6743000745773315 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.479549765586853 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6150804758071899 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1371985226869583 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010043350979685783 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6309456825256348 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7410784959793091 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13812433183193207 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9922407865524292 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17824074625968933 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7232633233070374 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14094148576259613 particle/accuracy_3_of_5‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898862361907959 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16502133011817932 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09849347174167633 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7691681385040283 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 _step‚779 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41002869606018066 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12538640201091766 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8214285373687744 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037433426827192307 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.031292520463466644 +jet/accuracy_1_of_1‚0.30309587717056274 +particle/specificity‚0.9113221764564514 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8273504376411438 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9946743249893188 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5435090661048889 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2858682870864868 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6062811613082886 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_2_of_4‚0.787034809589386 'particle/f_score‚0.6513764262199402 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16298796236515045 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7045790553092957 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39840349555015564 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13958343863487244 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15828731656074524 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 +jet/accuracy_3_of_4‚0.09573765844106674 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 epoch‚11 (loss/val_total_loss‚9.09969343864925 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6059672236442566 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9059392809867859 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9821584820747375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7264662981033325 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6874260902404785 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.031292520463466644 /particle/accuracy_3_of_4‚0.9965986609458923 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17696163058280945 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047915101051331 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15341684222221375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8529885411262512 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5524417161941528 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962959289550781 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_3_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8149122595787048 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9853572845458984 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 +jet/accuracy_2_of_2‚0.14053361117839813 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1073177233338356 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6250393390655518 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004882104694843292 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03244419023394585 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_5_of_5‚0 % _timestamp‚1.7845008081924815e+09 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8034361004829407 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9643168449401855 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.622542679309845 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17824074625968933 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026051048189401627 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6250000596046448 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5576664209365845 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6301707029342651 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7388700842857361 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6678752899169922 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08819051831960678 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18428996205329895 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 jet/accuracy_1_of_5‚1 _runtime‚3882.103883852 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932602643966675 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7248427271842957 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032626572996377945 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47183653712272644 *jet/accuracy_3_of_3‚0.0733180046081543 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6150804758071899 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7327925562858582 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5325521230697632 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032‹$€ðigç‚(:Ü ˤõÒ¹’¹+ network.recv‚ 1.145297e+06 disk.md0.out‚ 113.9765625 disk.dm-0.in‚ 19.68359375 disk.nvme0n1.in‚ 16.43359375" disk.nvme0n1.out‚ 43461.7109375 disk.nvme1c1n1.in‚3.25( proc.memory.availableMB‚ 134.26171875 network.sent‚ 1.2832093e+07( disk./.usagePercent‚76.2198183090441% disk./.usageGB‚1170.1647605895996 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 16.43359375 cpu‚26.29748381677497& disk.nvme1n1.out‚22812.97119140625 disk.md0.in‚0" disk.dm-0.out‚63061.6943359375 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 43461.7109375$ memory_percent‚99.18053150177002* proc.memory.percent‚17.381882667541504 proc.cpu.threads‚17 disk.nvme1n1.in‚3.25" disk.dm-1.out‚2969.52490234375 disk.rbd0.out‚0( disk.nvme1c1n1.out‚22812.97119140625# proc.memory.rssMB‚ 2847.84765625 %xw›è‚(: ˤõÒòþ˜, gpu.0.gpu‚72.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚161.044' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚53.681333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´—n2é- _wandbruntime‚3897  _runtime‚3897sH˜ÇêÁ _runtime‚3897.204382412 _step‚780 & lr-AdamW/pg1‚8.952225956929012e-05 & lr-AdamW/pg2‚8.952225956929012e-05 trainer/global_step‚9649 % _timestamp‚1.7845008232925577e+09KôÌëÆ _step‚780 & lr-AdamW/pg1‚8.952225956929012e-05 & lr-AdamW/pg2‚8.952225956929012e-05 trainer/global_step‚9649 % _timestamp‚1.7845008232925577e+09 _runtime‚3897.204382412Œ(æ<±2ì- _wandbruntime‚3897  _runtime‚3897–g×'íÑO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6795300679680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355579857064960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372266808116224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171388793 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6774090145834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367446573982576 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚379062213857280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6795300679680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73501255 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372266808116224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚9649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73501253 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171378648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24816944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362573218665984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379062213857280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 $ _timestamp‚1.784500823293314e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372266825155584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97887538 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97887538 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379062108795904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67346342 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42529644 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6795388701696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42529647 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355579881968640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6993149992448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367446591021936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379062108795904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374220767316244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6795388701696 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73501255 _runtime‚3897.204446043 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67346591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚248166989ÓÙÂxîÖO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362573007057408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97887538 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379062108795904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67346342 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42529644 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6795388701696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42529647 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355579881968640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6993149992448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367446591021936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379062108795904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374220767316244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6795388701696 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73501255 _runtime‚3897.204446043 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67346591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24816698 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6795300679680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355579857064960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372266808116224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171388793 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6774090145834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367446573982576 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚379062213857280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6795300679680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73501255 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372266808116224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚9649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73501253 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171378648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24816944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362573218665984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379062213857280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 $ _timestamp‚1.784500823293314e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372266825155584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97887538 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372304174483456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73508576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97887293 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _runtime‚3897.473615749 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379100258341376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171395867 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73508574 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374258315335560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171406026 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6795978789376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171395867 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6774766097278 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67353077 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367483463089696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367483446050336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 % _timestamp‚1.7845008235625262e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24819143 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374258212147614 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24819403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379100258341376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355615552222208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372304191522816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171406026 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372304174483456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6796066818560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360~ŽY‡ç&ñáM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379100258341376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171395867 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73508574 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374258315335560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171406026 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6795978789376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171395867 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6774766097278 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67353077 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367483463089696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367483446050336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 % _timestamp‚1.7845008235625262e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24819143 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374258212147614 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24819403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379100258341376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355615552222208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372304191522816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171406026 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372304174483456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6796066818560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362609397329920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97897450 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97897450 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362609606834176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚782 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6795978789376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67353340 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372304174483456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73508576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97887293 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _runtime‚3897.473615749 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208Ž£¹V2ò- _wandbruntime‚3897  _runtime‚3897%¨ró 1loss/FBt2/assignment_loss‚0.06579377454532596 epoch‚12 trainer/global_step‚9649 0loss/FRt1/assignment_loss‚2.0988846367405305 0loss/FRt2/assignment_loss‚0.9156129669863731 2loss/SRqqt2/assignment_loss‚0.6468941143824098 1loss/FBt1/assignment_loss‚0.00459801783952211 /loss/FRt1/detection_loss‚0.5311213064498002 1loss/SRqqt2/detection_loss‚0.6337920377875368 /loss/FBt1/detection_loss‚0.7930691734460803 $ _timestamp‚1.784500823563517e+09 %loss/total_loss‚9.527259197848176 _runtime‚3897.474132516 2loss/SRqqt1/assignment_loss‚0.6313795930208189 /loss/FRt2/detection_loss‚1.4133258549773018 1loss/SRqqt1/detection_loss‚1.0717948869513028 /loss/FBt2/detection_loss‚0.7209928347211745 _step‚783@š° ô† 1loss/SRqqt1/detection_loss‚1.0717948869513028 /loss/FBt2/detection_loss‚0.7209928347211745 _step‚783 1loss/FBt2/assignment_loss‚0.06579377454532596 epoch‚12 trainer/global_step‚9649 0loss/FRt1/assignment_loss‚2.0988846367405305 0loss/FRt2/assignment_loss‚0.9156129669863731 2loss/SRqqt2/assignment_loss‚0.6468941143824098 1loss/FBt1/assignment_loss‚0.00459801783952211 /loss/FRt1/detection_loss‚0.5311213064498002 1loss/SRqqt2/detection_loss‚0.6337920377875368 /loss/FBt1/detection_loss‚0.7930691734460803 $ _timestamp‚1.784500823563517e+09 %loss/total_loss‚9.527259197848176 _runtime‚3897.474132516 2loss/SRqqt1/assignment_loss‚0.6313795930208189 /loss/FRt2/detection_loss‚1.4133258549773018fv"~jõ‚(:ß Ú¤õÒª Ø+) disk./.usagePercent‚76.23624517443591 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0% disk.nvme1n1.out‚22813.6552734375 network.sent‚ 1.2902942e+07 network.recv‚ 1.149031e+06! disk.dm-0.out‚63062.701171875! disk.dm-1.out‚2969.6552734375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.43359375 disk.nvme1n1.in‚3.25( proc.memory.availableMB‚ 134.43359375! disk.nvme0c0n1.in‚ 16.43359375& disk.nvme0c0n1.out‚43463.123046875 disk.nvme1c1n1.in‚3.25' disk.nvme1c1n1.out‚22813.6552734375 disk.md0.out‚ 114.43359375* proc.memory.percent‚17.381882667541504 cpu‚26.295956691854045 proc.cpu.threads‚17$ disk.nvme0n1.out‚43463.123046875% disk./.usageGB‚1170.4169540405273 disk.rbd0.out‚0 disk.md0.in‚0$ memory_percent‚99.17948246002197# proc.memory.rssMB‚ 2847.84765625µ °6šö‚(: Ú¤õÒÿ‰ˆ, gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚196.653' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.55099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Âh÷‚(:Ý é¤õÒ­²Ô+ cpu‚26.29597256048384& disk.nvme0c0n1.out‚46331.623046875 disk.nvme1c1n1.in‚3.25' disk.nvme1c1n1.out‚22901.5380859375 disk.md0.out‚ 115.09765625# memory_percent‚99.1795301437378 network.sent‚ 1.2953183e+07% disk./.usageGB‚1174.2070693969727! disk.dm-1.out‚2969.7880859375 disk.md0.in‚0 proc.cpu.threads‚17 network.recv‚ 1.153563e+06) disk./.usagePercent‚76.48311801966928 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 16.43359375% disk.nvme1n1.out‚22901.5380859375! disk.dm-0.out‚66017.544921875! disk.nvme0c0n1.in‚ 16.43359375$ disk.nvme0n1.out‚46331.623046875 disk.nvme1n1.in‚3.25( proc.memory.availableMB‚ 134.42578125# proc.memory.rssMB‚ 2847.84765625* proc.memory.percent‚17.381882667541504DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380964364899328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6829326443520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73867303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380964364899328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364392941281792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_t£¼”µÝrain_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6829238421504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172241450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380964469960704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374135143517184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6807919621242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3917.463059096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6808005769716 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374135126477824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98384290 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6829326443520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376098096991818 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚9699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380964469960704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172251595% ÒÜ'þÖO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369290177370576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374135126477824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67684309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24939903 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6829238421504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172241450 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380964469960704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374135143517184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6807919621242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3917.463059096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6808005769716 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374135126477824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98384290 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6829326443520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376098096991818 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚9699 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380964469960704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172251595 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42744147 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364392738061824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357364614927360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380964364899328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6829326443520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73867303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380964364899328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364392941281792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚786 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67690731 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6829916531200 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24942562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67690999 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98394202 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6808595572686 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172258669 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357400334988288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845008438287408e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172268828 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6829916531200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚381002514444800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚374172492845056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6830004560384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42748434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376135645011022 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6808681721272 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚374172509884416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24942297 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357400310084608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚381002514444800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚374172492845056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172268828 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98384045jo÷áM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172268828 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98384045 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6830004560384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364429126282240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7029007089664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚786 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67690731 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6829916531200 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24942562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67690999 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98394202 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6808595572686 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172258669 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357400334988288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845008438287408e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172268828 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6829916531200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚381002514444800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚374172492845056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6830004560384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42748434 TNûmï× FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376135645011022 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6808681721272 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚374172509884416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24942297 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357400310084608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚381002514444800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚374172492845056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0’Áû¦2‚- _wandbruntime‚3917  _runtime‚3917 ÞZRƒ€ /loss/FBt2/detection_loss‚0.4793283611589756 0loss/FRt2/assignment_loss‚1.0120290992394934 1loss/SRqqt1/assignment_loss‚0.687676756868093 /loss/FRt2/detection_loss‚1.4440345732126174 1loss/SRqqt1/detection_loss‚1.1633410065321985 trainer/global_step‚9699 % _timestamp‚1.7845008438297791e+09 1loss/SRqqt2/assignment_loss‚0.570881469564861 1loss/FBt2/assignment_loss‚0.06163459253206845 %loss/total_loss‚9.737519760431917 epoch‚12 0loss/FRt1/assignment_loss‚2.1236689484725564 2loss/FBt1/assignment_loss‚0.024730020247757237 0loss/SRqqt2/detection_loss‚0.667431344449917 /loss/FBt1/detection_loss‚0.9410476943850518 _runtime‚3917.740441677 /loss/FRt1/detection_loss‚0.5617158937683274 _step‚787«³¶ „… _step‚787 /loss/FBt2/detection_loss‚0.4793283611589756 0loss/FRt2/assignment_loss‚1.0120290992394934 1loss/SRqqt1/assignment_loss‚0.687676756868093 /loss/FRt2/detection_loss‚1.4440345732126174 1loss/SRqqt1/detection_loss‚1.1633410065321985 trainer/global_step‚9699 % _timestamp‚1.7845008438297791e+09 1loss/SRqqt2/assignment_loss‚0.570881469564861 1loss/FBt2/assignment_loss‚0.06163459253206845 %loss/total_loss‚9.737519760431917 epoch‚12 0loss/FRt1/assignment_loss‚2.1236689484725564 2loss/FBt1/assignment_loss‚0.024730020247757237 0loss/SRqqt2/detection_loss‚0.667431344449917 /loss/FBt1/detection_loss‚0.9410476943850518 _runtime‚3917.740441677 /loss/FRt1/detection_loss‚0.5617158937683274“.û–a…‚(:Ö ø¤õÒžÇï+' disk.nvme1c1n1.out‚22904.9775390625& proc.memory.availableMB‚ 134.109375" proc.memory.rssMB‚ 2847.8515625 network.sent‚ 1.2969049e+07 disk.rbd0.in‚0 disk.nvme1c1n1.in‚3.25$ memory_percent‚99.18146133422852* proc.memory.percent‚17.381906509399414 proc.cpu.threads‚17 network.recv‚ 1.156627e+06$ disk./.usageGB‚1171.725570678711 disk.nvme1n1.in‚3.25 disk.dm-1.out‚2970.400390625 disk.nvme0n1.in‚ 16.43359375$ disk.nvme0n1.out‚47172.919921875 cpu‚26.2990916030814) disk./.usagePercent‚76.32148319027591 disk.md0.out‚ 116.28125 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0& disk.nvme0c0n1.out‚47172.919921875% disk.nvme1n1.out‚22904.9775390625 disk.md0.in‚0" disk.dm-0.out‚66859.2333984375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375; hž†‚(:„ ø¤õÒ¢î, gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.268' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔV‚C2‡- _wandbruntime‚3938  _runtime‚3938WyºÑLjÁ & lr-AdamW/pg2‚8.925611850791363e-05 trainer/global_step‚9749 % _timestamp‚1.7845008641489353e+09 _runtime‚3938.059959404 _step‚788 & lr-AdamW/pg1‚8.925611850791363e-05ÓÆ”4Ì‰Æ % _timestamp‚1.7845008641489353e+09 _runtime‚3938.059959404 _step‚788 & lr-AdamW/pg1‚8.925611850791363e-05 & lr-AdamW/pg2‚8.925611850791363e-05 trainer/global_step‚9749”V¢Î=2Š- _wandbruntime‚3938  _runtime‚3938á?_Ç×'‹ÑO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25063918 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74233355 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6863176163328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376003444839424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98881042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25064168 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98870899 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376003461878784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98870899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42958644 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359149372789760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371133797797936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784500864151167e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382866726064128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74233353 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚377975633043060 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚789 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173104252 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382866726064128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6841835245124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382866621002752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7063074822656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98881042 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3938.062229369 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382866621002752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9749 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74233355 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6863176163328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359149397693440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚377975529855226 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6863264185344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68022562 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173114397 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68022815 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74233353 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42958647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104#4mÜ'ŒÖO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6841835245124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382866621002752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7063074822656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98881042 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚3938.062229369 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382866621002752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9749 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74233355 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6863176163328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359149397693440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚377975529855226 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6863264185344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68022562 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173114397 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68022815 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74233353 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42958647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25063918 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74233355 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6863176163328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376003444839424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98881042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25064168 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98870899 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376003461878784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98870899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42958644 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359149372789760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371133797797936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784500864151167e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382866726064128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74233353 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚377975633043060 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚789 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173104252 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382866726064128•­c¤'2- _wandbruntime‚3938  _runtime‚3938qÓiº¸ŽÛM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376040811206656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6863942302208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366249050227712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68029237 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42962934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25066554 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25066303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6842425048094 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚9749 % _timestamp‚1.7845008644275753e+09 _runtime‚ 3938.33871401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359185092850688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98890954 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6842511196680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6863942302208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98890954 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173121471 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173131630 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6863854273024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98880797 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74240676 ;4Deviceúƒœ)StatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68029491 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7063770679296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173121471 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7063957377024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382904770548224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚371170669865696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376040828246016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74240674 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173131630 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376040828246016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359185067947008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376040811206656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371170652826336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382904770548224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42962937 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366248838626304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6863854273024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378013181062376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74240674 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0¦U1æ&àM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376040811206656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6863942302208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366249050227712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68029237 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42962934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25066554 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25066303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6842425048094 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚9749 % _timestamp‚1.7845008644275753e+09 _runtime‚ 3938.33871401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359185092850688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98890954 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6842511196680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6863942302208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98890954 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173121471 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173131630 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6863854273024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98880797 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74240676 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68029491 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7063770679296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173121471 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7063957377024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382904770548224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚371170669865696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376040828246016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74240674 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173131630 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376040828246016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359185067947008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376040811206656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371170652826336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382904770548224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42962937 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366248838626304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6863854273024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378013181062376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74240674 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0–RKWÁ2- _wandbruntime‚3938  _runtime‚3938j€õ ‘ƒ 0loss/FRt2/assignment_loss‚0.9730498907612819 2loss/FBt1/assignment_loss‚0.006917970610360041 /loss/FRt1/assignment_loss‚2.325220838803273 1loss/SRqqt1/detection_loss‚1.0138129716529356 epoch‚12 % _timestamp‚1.7845008644290738e+09 _runtime‚3938.339726702 _step‚791 2loss/SRqqt1/assignment_loss‚0.7247080837429613 3loss/FBt2/assignment_loss‚0.0005999503604747984 .loss/FRt2/detection_loss‚1.483264077582634 1loss/SRqqt2/detection_loss‚0.6394655261874983 /loss/FBt1/detection_loss‚0.7646848978031249 /loss/FBt2/detection_loss‚0.8649645972275927 2loss/SRqqt2/assignment_loss‚0.6220227009258034 /loss/FRt1/detection_loss‚0.5488636173922044 %loss/total_loss‚9.967575123050143 trainer/global_step‚9749¦â ’ˆ % _timestamp‚1.7845008644290738e+09 _runtime‚3938.339726702 _step‚791 2loss/SRqqt1/assignment_loss‚0.7247080837429613 3loss/FBt2/assignment_loss‚0.0005999503604747984 .loss/FRt2/detection_loss‚1.483264077582634 1loss/SRqqt2/detection_loss‚0.6394655261874983 /loss/FBt1/detection_loss‚0.7646848978031249 /loss/FBt2/detection_loss‚0.8649645972275927 2loss/SRqqt2/assignment_loss‚0.6220227009258034 /loss/FRt1/detection_loss‚0.5488636173922044 %loss/total_loss‚9.967575123050143 trainer/global_step‚9749 0loss/FRt2/assignment_loss‚0.9730498907612819 2loss/FBt1/assignment_loss‚0.006917970610360041 /loss/FRt1/assignment_loss‚2.325220838803273 1loss/SRqqt1/detection_loss‚1.0138129716529356 epoch‚12—°èÎÞc“‚(:Ø ‡¥õÒóØ+$ memory_percent‚99.18015003204346 cpu‚26.3014345740459 proc.cpu.threads‚17' disk.nvme1c1n1.out‚22911.1513671875 disk.md0.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 134.32421875* proc.memory.percent‚17.381906509399414 network.sent‚ 1.3031291e+07$ disk.nvme0c0n1.out‚ 47194.4765625" disk.nvme0n1.out‚ 47194.4765625 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.md0.out‚ 116.70703125" proc.memory.rssMB‚ 2847.8515625) disk./.usagePercent‚76.32110600672604 disk.nvme0n1.in‚ 16.43359375% disk.nvme1n1.out‚22911.1513671875 disk.dm-0.in‚ 19.68359375" disk.dm-0.out‚66885.8779296875 disk.dm-1.out‚2970.576171875 network.recv‚ 1.160663e+06% disk./.usageGB‚1171.7197799682617! disk.nvme0c0n1.in‚ 16.43359375 disk.rbd0.in‚0 disk.rbd0.out‚0¸cöñš”‚(: ‡¥õÒâñ…, gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚199.289' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.42966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žp›¾2•- _wandbruntime‚3958  _runtime‚3958Q£"iÅ–¿ & lr-AdamW/pg1‚8.912212487101409e-05 & lr-AdamW/pg2‚8.912212487101409e-05 trainer/global_step‚9799 # _timestamp‚1.78450088474743e+09 _runtime‚3958.658493258 _step‚792 tiïÊ—Ä & lr-AdamW/pg1‚8.912212487101409e-05 & lr-AdamW/pg2‚8.912212487101409e-05 trainer/global_step‚9799 # _timestamp‚1.78450088474743e+09 _runtime‚3958.658493258 _step‚792˜ „×À2˜- _wandbruntime‚3958  _runtime‚3958° ¾Ø'™ÒO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74599405 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173977199 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74599403 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99377794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377871763201024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43173144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7098191660032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7098011246592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875578572058 trainer/global_step‚9799 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25188050 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173967054 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384768982167552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377871780240384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384768877106176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897113905152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68360944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚368032347215872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99367651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25187800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3958.660444871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43173147 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372977401185936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330ƒÓ_\…š×O QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875578572058 trainer/global_step‚9799 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25188050 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173967054 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384768982167552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377871780240384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reservepå©\Xd_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384768877106176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897113905152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68360944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚368032347215872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99367651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25187800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3958.660444871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43173147 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372977401185936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74599405 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173977199 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74599403 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99377794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377871763201024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43173144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7098191660032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7098011246592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0™xóNŠ2›- _wandbruntime‚3958  _runtime‚3958½éjâ&œÜM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360969825809408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74606724 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99377549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68367734 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74606726 trainer/global_step‚9799 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43177437 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25190300 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6876254523502 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74606726 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99377549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6897792014848 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384807026651648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚384806921583104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377909146607616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3958.916611652 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377909129568256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74606724 % _timestamp‚1.7845008850055397e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99387706 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379890613925784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25190567 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7098708299264 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173994432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6897880044032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360969850713088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚794 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384807026651648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377909146607616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373014256214336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6897880044032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173984273Ïå£ç&áM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6876254523502 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74606726 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99377549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6897792014848 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384807026651648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚384806921583104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377909146607616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3958.916611652 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377909129568256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74606724 % _timestamp‚1.7845008850055397e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99387706 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379890613925784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25190567 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7098708299264 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173994432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6897880044032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360969850713088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚794 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384807026651648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377909146607616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373014256214336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6897880044032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173984273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360969825809408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74606724 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99377549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68367734 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74606726 trainer/global_step‚9799 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43177437 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25190300šäjåU2ž- _wandbruntime‚3958  _runtime‚3958¨LgÔ Ÿ„ .loss/FRt2/detection_loss‚1.434134394955264 1loss/SRqqt1/detection_loss‚0.9975267076020286 0loss/FBt2/detection_loss‚0.40001886640327855 trainer/global_step‚9799 _step‚795 0loss/FRt2/assignment_loss‚0.9924323769458326 2loss/SRqqt1/assignment_loss‚0.5253419713381106 2loss/SRqqt2/assignment_loss‚0.6199973323906472 2loss/FBt1/assignment_loss‚0.008853879613212712 1loss/SRqqt2/detection_loss‚0.6096728287273726 _runtime‚3958.917099012 %loss/total_loss‚9.137391619988312 epoch‚12 % _timestamp‚1.7845008850064304e+09 0loss/FRt1/assignment_loss‚2.1833474233343795 2loss/FBt2/assignment_loss‚0.002703373803907442 /loss/FBt1/detection_loss‚0.8267682197503745 /loss/FRt1/detection_loss‚0.5365942451239042oÌ]¿ ‰ /loss/FRt1/detection_loss‚0.5365942451239042 .loss/FRt2/detection_loss‚1.434134394955264 1loss/SRqqt1/detection_loss‚0.9975267076020286 0loss/FBt2/detection_loss‚0.40001886640327855 trainer/global_step‚9799 _step‚795 0loss/FRt2/assignment_loss‚0.9924323769458326 2loss/SRqqt1/assignment_loss‚0.5253419713381106 2loss/SRqqt2/assignment_loss‚0.6199973323906472 2loss/FBt1/assignment_loss‚0.008853879613212712 1loss/SRqqt2/detection_loss‚0.6096728287273726 _runtime‚3958.917099012 %loss/total_loss‚9.137391619988312 epoch‚12 % _timestamp‚1.7845008850064304e+09 0loss/FRt1/assignment_loss‚2.1833474233343795 2loss/FBt2/assignment_loss‚0.002703373803907442 /loss/FBt1/detection_loss‚0.8267682197503745›èãPè¡‚(:„ –¥õÒÑǵ, gpu.0.gpu‚56.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.948' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iñžåa¢‚(:Ö –¥õÒÑë‹-$ disk./.usageGB‚1171.717903137207 disk.dm-1.in‚0' disk.nvme0c0n1.out‚47200.6474609375% disk.nvme0n1.out‚47200.6474609375 disk.md0.out‚117.125$ memory_percent‚99.18186664581299" proc.memory.rssMB‚ 2847.8515625 cpu‚26.3007175686822 network.recv‚ 1.164477e+06) disk./.usagePercent‚76.32098375751227 disk.rbd0.out‚0 disk.nvme0n1.in‚ 16.43359375( proc.memory.availableMB‚ 134.04296875 disk.dm-1.out‚2970.712890625 disk.rbd0.in‚0& disk.nvme1c1n1.out‚22912.552734375 disk.nvme1n1.in‚3.25$ disk.nvme1n1.out‚22912.552734375 disk.dm-0.in‚ 19.68359375* proc.memory.percent‚17.381906509399414 network.sent‚ 1.3080645e+07! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 disk.md0.in‚0" disk.dm-0.out‚66892.4013671875 proc.cpu.threads‚17Â/7‹f£‚(:Û ¥¥õÒ±ÊÏ+ disk.rbd0.out‚0$ memory_percent‚99.18239116668701" proc.memory.rssMB‚ 2847.8515625 cpu‚26.302678444481526) disk./.usagePercent‚76.20818376191721$ disk./.usageGB‚1169.986141204834! disk.nvme0c0n1.in‚ 16.43359375# disk.nvme0n1.out‚47206.81640625' disk.nvme1c1n1.out‚22916.1708984375" disk.dm-0.out‚66900.8466796875 disk.dm-1.out‚2970.841796875( proc.memory.availableMB‚ 133.95703125 network.recv‚ 1.168211e+06 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25% disk.nvme1n1.out‚22916.1708984375 disk.md0.out‚ 117.70703125 disk.dm-0.in‚ 19.68359375* proc.memory.percent‚17.381906509399414 proc.cpu.threads‚17 network.sent‚ 1.3143924e+07% disk.nvme0c0n1.out‚47206.81640625 disk.nvme0n1.in‚ 16.43359375 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0wßX蜤‚(:‘ ¥¥õÒŸ³‚, gpu.0.gpu‚87.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚169.596' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚56.532000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øö2¥- _wandbruntime‚3977  _runtime‚3977¾´1áƦÀ trainer/global_step‚9849 % _timestamp‚1.7845009039960592e+09 _runtime‚ 3977.90838674 _step‚796 & lr-AdamW/pg1‚8.898751868826524e-05 & lr-AdamW/pg2‚8.898751868826524e-05Ó6úä˧Š_step‚796 & lr-AdamW/pg1‚8.898751868826524e-05 & lr-AdamW/pg2‚8.898751868826524e-05 trainer/global_step‚9849 % _timestamp‚1.7845009039960592e+09 _runtime‚ 3977.90838674œ…è´ˆ2¨- _wandbruntime‚3977  _runtime‚3977Þ³­ä ©ÐO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362718913418240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386671238270976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6931051646976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379740081562624 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386671133209600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379740081562624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74965455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6931139668992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6909494195940 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386671133209600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379740098601984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6931139668992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386671238270976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99864403 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚797 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174829856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99874546 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381730395582042 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚374821004573936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6909408047466 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99864403 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 $ _timestamp‚1.784500903996614e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74965455 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25312300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6931051646976!¤äýÛ'ªÕO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6931051646976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362718913418240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386671238270976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6931051646976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379740081562624 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386671133209600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379740081562624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74965455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6931139668992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6909494195940 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386671133209600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379740098601984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6931139668992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386671238270976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99864403 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚797 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174829856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99874546 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381730395582042 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚374821004573936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6909408047466 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99864403 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 $ _timestamp‚1.784500903996614e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74965455 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25312300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Aò .2«- _wandbruntime‚3978  _runtime‚3978ôõâ&¬ÜM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6931729756672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99884458 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379777447929856 _runtime‚3978.162931903 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174857234 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362754583671808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6931729756672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74972776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚381768046789192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74972774 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99884458 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386709177686528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379777447929856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6931817785856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174847075 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99874301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25314718 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚381767943601246 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚9849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6931817785856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68706414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369888252846592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362754608575488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7133849581056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174857234 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174847075 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7133669174784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6910083998910 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚374857859602336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99874301 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379777464969216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚798 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68706655 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74972776<±ø'#­áM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99884458 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379777447929856 _runtime‚3978.162931903 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174857234 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362754583671808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6931729756672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74972776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚381768046789192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74972774 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99884458 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386709177686528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379777447929856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6931817785856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174847075 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99874301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25314718 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚381767943601246 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚9849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6931817785856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68706414 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369888252846592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362754608575488 aODeviceStaȳkMÄtsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7133849581056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174857234 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174847075 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7133669174784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6910083998910 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚374857859602336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99874301 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379777464969216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚798 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68706655 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74972776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6931729756672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912žÝk¦ñ2®- _wandbruntime‚3978  _runtime‚3978º‘W⯂ /loss/FRt1/detection_loss‚0.5947883977338833 1loss/SRqqt1/detection_loss‚1.0810412360737338 epoch‚12 trainer/global_step‚9849 % _timestamp‚1.7845009042527838e+09 2loss/SRqqt2/assignment_loss‚0.5645270848264283 .loss/FRt2/detection_loss‚1.523709794167768 _runtime‚3978.163437104 _step‚799 1loss/SRqqt1/assignment_loss‚0.766407945917961 3loss/FBt2/assignment_loss‚0.0056957571829155725 /loss/FBt1/detection_loss‚0.7261928883583649 1loss/SRqqt2/detection_loss‚0.7278847653460481 /loss/FBt2/detection_loss‚0.5587394413448972 $loss/total_loss‚9.54319029358857 0loss/FRt1/assignment_loss‚2.1118373126757066 0loss/FRt2/assignment_loss‚0.8617682698344397 2loss/FBt1/assignment_loss‚0.020597400126422617AÎÕx °‡ % _timestamp‚1.7845009042527838e+09 2loss/SRqqt2/assignment_loss‚0.5645270848264283 .loss/FRt2/detection_loss‚1.523709794167768 _runtime‚3978.163437104 _step‚799 1loss/SRqqt1/assignment_loss‚0.766407945917961 3loss/FBt2/assignment_loss‚0.0056957571829155725 /loss/FBt1/detection_loss‚0.7261928883583649 1loss/SRqqt2/detection_loss‚0.7278847653460481 /loss/FBt2/detection_loss‚0.5587394413448972 $loss/total_loss‚9.54319029358857 0loss/FRt1/assignment_loss‚2.1118373126757066 0loss/FRt2/assignment_loss‚0.8617682698344397 2loss/FBt1/assignment_loss‚0.020597400126422617 /loss/FRt1/detection_loss‚0.5947883977338833 1loss/SRqqt1/detection_loss‚1.0810412360737338 epoch‚12 trainer/global_step‚9849Ÿê̯;e±‚(:Ú ´¥õÒÈÖõ+ disk.rbd0.in‚0 network.sent‚ 1.3159785e+07% disk./.usageGB‚1172.0562591552734 disk.nvme0n1.in‚ 16.43359375( disk.nvme1c1n1.out‚22916.75146484375 disk.nvme1n1.in‚3.25& disk.nvme1n1.out‚22916.75146484375$ disk.nvme0c0n1.out‚ 49307.0859375$ memory_percent‚99.18274879455566( disk./.usagePercent‚76.3430229054078 disk.nvme1c1n1.in‚3.25 disk.dm-1.in‚0' proc.memory.availableMB‚ 133.8984375* proc.memory.percent‚17.381906509399414 cpu‚26.301665135927276 proc.cpu.threads‚17 network.recv‚ 1.171705e+06" disk.nvme0n1.out‚ 49307.0859375 disk.md0.out‚ 117.9453125! disk.dm-0.out‚69001.041015625 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375" proc.memory.rssMB‚ 2847.8515625 disk.md0.in‚0 disk.dm-0.in‚ 19.68359375" disk.dm-1.out‚2970.98779296875ÕwÌŽ›²‚(: ´¥õÒý¹¢, gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚112.391' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.46366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬9¶2³- _wandbruntime‚3997  _runtime‚3997¤ ÆÁÆ´À & lr-AdamW/pg1‚8.885230211565833e-05 & lr-AdamW/pg2‚8.885230211565833e-05 trainer/global_step‚9899 % _timestamp‚1.7845009232438703e+09 _runtime‚ 3997.15482133 _step‚800 i%¿ËµÅ & lr-AdamW/pg1‚8.885230211565833e-05 & lr-AdamW/pg2‚8.885230211565833e-05 trainer/global_step‚9899 % _timestamp‚1.7845009232438703e+09 _runtime‚ 3997.15482133 _step‚800 0–’i2¶- _wandbruntime‚3997  _runtime‚3997W2$4Ü'·ÖO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69038245 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371671518388224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75331505 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6964989388800 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100371298 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6943323671348 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371671729996800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364503646376960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388573494374400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6965077410816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25435844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381608416963584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75331505 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175702803 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 % _timestamp‚1.7845009232447567e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381608399924224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388573389313024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364503671280640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚801 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚381608399924224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6943237522874 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100361155 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100361155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9899 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388573389313024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100371298 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25436098 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175692658 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6964989388800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7168058716160 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376664590922576 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6965077410816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75331503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7167872011264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383607931633284 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388573494374400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383607828445450ER±á'¸ÛO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75331505 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6964989388800 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100371298 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6943323671348 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371671729996800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364503646376960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388573494374400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6965077410816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25435844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381608416963584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75331505 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175702803 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 % _timestamp‚1.7845009232447567e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381608399924224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388573389313024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364503671280640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚801 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚381608399924224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6943237522874 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100361155 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100361155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9899 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388573389313024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100371298 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25436098 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175692658 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6964989388800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7168058716160 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376664590922576 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6965077410816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75331503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7167872011264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383607931633284 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388573494374400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383607828445450 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69038245 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371671518388224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¡Õ:Ò2¹- _wandbruntime‚3997  _runtime‚39978k?F¯ºàM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6965755527680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100371053 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7168572459520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376701462990336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 trainer/global_step‚9899 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6965667498496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 _runtime‚3997.412755787 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75338824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371708121400832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75338826 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43606437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376701480029696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6943913474318 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100371053 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6965755527680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175720036 cODeviceStatsMonitor.on_train_batch_end/inactive_split_byt46DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75338826 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381645766291456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25438312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175709877 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6965667498496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100381210 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69044746 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383645376464654 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175720036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25438557 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6965755527680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100371053 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7168572459520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376701462990336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 trainer/global_step‚9899 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6965667498496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 _runtime‚3997.412755787 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75338824 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371708121400832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75338826 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43606437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376701480029696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6943913474318 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100371053 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6965755527680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175720036 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364539366437888 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75338826 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381645766291456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25438312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175709877 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6965667498496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100381210 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69044746 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383645376464654 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175720036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25438557 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6977153146756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383476735325184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6977066998282 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69375626 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚366288429143040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6998927130624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75697553 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69375897 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7202762883584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390475645416448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚385485261308858 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100868050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚385485364496692 _runtime‚ 4015.55935896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43816647 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100857907 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176555460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383476718285824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚9949 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100857907 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75697553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390475750477824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383476735325184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43816644 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390475645416448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚385485261308858 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100868050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚385485364496692 _runtime‚ 4015.55935896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43816647 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100857907 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176555460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383476718285824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚9949 M?DeviceStatsMonitor.on_train_batch_|M‚start/active.small_pool.freed‚ 100857907 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75697553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390475750477824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383476735325184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43816644 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6977153146756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383476735325184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6977066998282 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69375626¥Tí(2Ç- _wandbruntime‚4015  _runtime‚4015ùâÊLæ&ÈàM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚390513794961920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6999693269504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7203469160448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6999605240320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚385522809328062 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378545066378336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6999605240320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75704876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚4015.811989792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378545083417696 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6977829098312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43820934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383514101692416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383514101692416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75704876 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100877962 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100867805 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69382326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43820937 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚366324099396608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7203643275264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176582838 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚385522912516008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25561392 % _timestamp‚1.7845009419008956e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69382571 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383514084653056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75704874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383514084653056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100867805 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373527568557056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6977742949726 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚366324124300288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ñ¯ë&ÉåM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚385522912516008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25561392 % _timestamp‚1.7845009419008956e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69382571 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383514084653056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75704874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383514084653056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100867805 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373527568557056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6977742949726 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚366324124300288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚390513794961920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6999693269504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7203469160448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6999605240320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚385522809328062 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378545066378336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6999605240320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75704876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚4015.811989792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378545083417696 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6977829098312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43820934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383514101692416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383514101692416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75704876 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100877962 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100867805 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69382326 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43820937 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚366324099396608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7203643275264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176582838¦Ödf2Ê- _wandbruntime‚4015  _runtime‚4015’‚ú ˃ 2loss/SRqqt1/assignment_loss‚0.6344534372548252 2loss/SRqqt2/assignment_loss‚0.6458804817827783 /loss/FRt1/detection_loss‚0.6004682940349242 1loss/SRqqt1/detection_loss‚1.1462964149054566 /loss/FBt1/detection_loss‚0.8508617249982697 trainer/global_step‚9949 $ _timestamp‚1.784500941901825e+09 0loss/FRt2/assignment_loss‚1.0235146723175776 0loss/FBt1/assignment_loss‚0.1334694635058374 /loss/FRt2/detection_loss‚1.4574276410560443 epoch‚12 1loss/SRqqt2/detection_loss‚0.7672571125122517 0loss/FBt2/detection_loss‚0.49107440016691534 %loss/total_loss‚9.878467006895447 _step‚807 0loss/FRt1/assignment_loss‚2.1252069528774253 3loss/FBt2/assignment_loss‚0.0025564114831407186 _runtime‚4015.812438593ëÿÅ&̈ trainer/global_step‚9949 $ _timestamp‚1.784500941901825e+09 0loss/FRt2/assignment_loss‚1.0235146723175776 0loss/FBt1/assignment_loss‚0.1334694635058374 /loss/FRt2/detection_loss‚1.4574276410560443 epoch‚12 1loss/SRqqt2/detection_loss‚0.7672571125122517 0loss/FBt2/detection_loss‚0.49107440016691534 %loss/total_loss‚9.878467006895447 _step‚807 0loss/FRt1/assignment_loss‚2.1252069528774253 3loss/FBt2/assignment_loss‚0.0025564114831407186 _runtime‚4015.812438593 2loss/SRqqt1/assignment_loss‚0.6344534372548252 2loss/SRqqt2/assignment_loss‚0.6458804817827783 /loss/FRt1/detection_loss‚0.6004682940349242 1loss/SRqqt1/detection_loss‚1.1462964149054566 /loss/FBt1/detection_loss‚0.8508617249982697§N‹e`WÍ‚(:Ì Ò¥õÒüϲ+* proc.memory.percent‚17.381906509399414 cpu‚26.299366354563517 network.sent‚ 1.3272788e+07 disk.nvme1c1n1.out‚22920.25 disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.nvme0n1.in‚ 16.43359375 proc.cpu.threads‚17 disk.nvme1n1.out‚22920.25 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375' proc.memory.availableMB‚ 133.7109375 disk.nvme1c1n1.in‚3.25 disk.dm-0.in‚ 19.68359375# disk.dm-0.out‚70462.08154296875 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚50766.69873046875$ memory_percent‚99.18389320373535" proc.memory.rssMB‚ 2847.8515625 network.recv‚ 1.180334e+06) disk./.usagePercent‚76.32121309902512% disk./.usageGB‚1171.7214241027832 disk.md0.out‚ 118.71875 disk.dm-1.out‚ 2971.390625& disk.nvme0n1.out‚50766.69873046875Vÿ½…šÎ‚(: Ò¥õÒå‘, gpu.0.gpu‚66.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚183.866' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.28866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eè]}2Ï- _wandbruntime‚4034  _runtime‚4034NãïÆÐÀ & lr-AdamW/pg1‚8.858004647368376e-05 & lr-AdamW/pg2‚8.858004647368376e-05 trainer/global_step‚9999 % _timestamp‚1.7845009602778213e+09 _runtime‚ 4034.18873226 _step‚808ôéflËÑÅ % _timestamp‚1.7845009602778213e+09 _runtime‚ 4034.18873226 _step‚808 & lr-AdamW/pg1‚8.858004647368376e-05 & lr-AdamW/pg2‚8.858004647368376e-05 trainer/global_step‚9999¨šÀ®›2Ò- _wandbruntime‚4034  _runtime‚4034{s‘êÓÕO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚385345036647424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚385345036647424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385345053686784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392378006581248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101354659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.V¿­'ñsmall_pool.allocated‚ 101364802 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380351797698576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392377901519872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101354659 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69713567 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380351814737936 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101364802 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392377901519872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177428407 _step‚809 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177418262 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25682179 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784500960278647e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76063605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76063603 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387362694172266 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7237889180672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44031144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7010896473690 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25682420 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _runtime‚4034.189448721 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76063603 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375311076186112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7032864872448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032952894464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69713323 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032952894464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69713323 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101354659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101364802 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380351797698576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392377901519872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101354659 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69713567 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380351814737936 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101364802 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392377901519872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177428407 _step‚809 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177418262 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25682179 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784500960278647e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76063605 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76063603 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387362694172266 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152©aÄ2Õ- _wandbruntime‚4034  _runtime‚4034l¡›Éæ&ÖàM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7238402637312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387400345379416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380388669766336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚385382420054016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76070926 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44035437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385382420054016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76070926 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7033542982144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177445640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177435481 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101374714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385382403014656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101374714 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177445640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392415945996800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387400242191470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25684893 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380388686805696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44035434 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101364557 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7011572425134 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101364557 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385382403014656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69720094 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7238583043584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7033631011328 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69720330 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375347259896320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482h–}ô×åM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101364557 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385382403014656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69720094 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7238583043584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7033631011328 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.aþ"÷allocated‚69720330 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375347259896320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7238402637312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387400345379416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380388669766336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚385382420054016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76070926 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44035437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385382420054016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76070926 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7033542982144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177445640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177435481 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101374714 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385382403014656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101374714 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177445640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392415945996800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387400242191470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25684893 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380388686805696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44035434 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101364557 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7011572425134 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330ªãõˆÿ2Ø- _wandbruntime‚4034  _runtime‚4034²}úÙ‚ 3loss/FBt1/assignment_loss‚0.0007036608170197081 trainer/global_step‚9999 _runtime‚4034.445571733 2loss/SRqqt2/assignment_loss‚0.7152246377704062 1loss/SRqqt1/detection_loss‚1.1322906708588736 epoch‚12 2loss/SRqqt1/assignment_loss‚0.8408796690484797 1loss/FBt2/assignment_loss‚0.07148406372865104 /loss/FRt1/detection_loss‚0.5026485258963814 0loss/SRqqt2/detection_loss‚0.616239053997981 /loss/FBt1/detection_loss‚1.0890948533573572 /loss/FBt2/detection_loss‚0.5455157980322838 % _timestamp‚1.7845009605348256e+09 .loss/FRt1/assignment_loss‚2.17833402383426 0loss/FRt2/assignment_loss‚0.9229544233817321 /loss/FRt2/detection_loss‚1.4252831008786766 &loss/total_loss‚10.040652481602102 _step‚811À„ Ú‡ 2loss/SRqqt2/assignment_loss‚0.7152246377704062 1loss/SRqqt1/detection_loss‚1.1322906708588736 epoch‚12 2loss/SRqqt1/assignment_loss‚0.8408796690484797 1loss/FBt2/assignment_loss‚0.07148406372865104 /loss/FRt1/detection_loss‚0.5026485258963814 0loss/SRqqt2/detection_loss‚0.616239053997981 /loss/FBt1/detection_loss‚1.0890948533573572 /loss/FBt2/detection_loss‚0.5455157980322838 % _timestamp‚1.7845009605348256e+09 .loss/FRt1/assignment_loss‚2.17833402383426 0loss/FRt2/assignment_loss‚0.9229544233817321 /loss/FRt2/detection_loss‚1.4252831008786766 &loss/total_loss‚10.040652481602102 _step‚811 3loss/FBt1/assignment_loss‚0.0007036608170197081 trainer/global_step‚9999 _runtime‚4034.445571733«á‰cÛ‚(:Ø á¥õÒ§Ðá+' proc.memory.availableMB‚ 133.6171875 network.sent‚ 1.3347847e+07 disk.md0.out‚ 119.1796875 disk.dm-1.out‚ 2971.515625* proc.memory.percent‚17.381906509399414 proc.cpu.threads‚17 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375( disk.nvme0c0n1.out‚50777.74462890625" proc.memory.rssMB‚ 2847.8515625 network.recv‚ 1.184383e+06( disk./.usagePercent‚76.3211174365306 disk.nvme0n1.in‚ 16.43359375# memory_percent‚99.1844654083252 cpu‚26.29819029040185$ disk./.usageGB‚1171.719955444336& disk.nvme0n1.out‚50777.74462890625& disk.nvme1c1n1.out‚22924.919921875$ disk.nvme1n1.out‚22924.919921875 disk.dm-0.in‚ 19.68359375# disk.dm-0.out‚70476.69189453125 disk.rbd0.out‚0áKþ›Ü‚(: á¥õÒæÓŠ, gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.781' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.92700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ$T¬gÝ‚(:Ü ð¥õÒðûÞ+ disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.dm-0.in‚ 19.68359375 disk.rbd0.out‚0" disk.nvme0n1.out‚ 50782.6796875 proc.cpu.threads‚17 network.sent‚ 1.3398176e+07% disk./.usageGB‚1168.9639434814453 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚23019.02685546875 disk.nvme0n1.in‚ 16.43359375' proc.memory.availableMB‚ 133.5859375" proc.memory.rssMB‚ 2847.8515625* proc.memory.percent‚17.381906509399414& disk.nvme1n1.out‚23019.02685546875 disk.md0.out‚ 119.80859375! disk.dm-0.out‚70574.287109375" disk.dm-1.out‚2971.65771484375! disk.nvme0c0n1.in‚ 16.43359375$ memory_percent‚99.18465614318848 cpu‚26.296905323341132 network.recv‚ 1.188283e+06) disk./.usagePercent‚76.14160192030258 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 50782.6796875Ýr'µ›Þ‚(: ð¥õÒÑþˆ, gpu.0.gpu‚78.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚138.049' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚46.016333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸®Å(2ß- _wandbruntime‚4052  _runtime‚4052vóôÈà & lr-AdamW/pg2‚8.844301176504279e-05 trainer/global_step‚10049 % _timestamp‚1.7845009790516417e+09 _runtime‚4052.962528714 _step‚812 & lr-AdamW/pg1‚8.844301176504279e-05c\ÍáÇ _runtime‚4052.962528714 _step‚812 & lr-AdamW/pg1‚8.844301176504279e-05 & lr-AdamW/pg2‚8.844301176504279e-05 trainer/global_step‚10049 % _timestamp‚1.7845009790516417e+09¬ü88$2â- _wandbruntime‚4052  _runtime‚4052…"äiÝ'ã×O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76429655 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394280262684672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7066890636288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7272659475456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387213355009024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76429655 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101861554 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101851411 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76429653 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7044725949098 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚394280262684672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚382195418125936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70051958 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚377130791048192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76429653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7066802614272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394280157623296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7066802614272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚389240127035674 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369857919964160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101861554 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44245644 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70052193 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7272846180352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178291209 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101851411 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806546 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚377130579439616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394280157623296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44245647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚369857944867840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387213372048384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7272659475456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387213355009024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76429655 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101861554 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101851411 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76429653 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7044725949098 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚394280262684672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚382195418125936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70051958 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚377130791048192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76429653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7066802614272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394280157623296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7066802614272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚389240127035674 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369857919964160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101861554 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44245644 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70052193 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7272846180352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178291209 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101851411 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806546 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚377130579439616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394280157623296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44245647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚369857944867840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387213372048384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387250738415616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 EDeviceStatsMonitor.on_train_batch_end/reserved_byt ©7es.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚382232273154336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚814 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7045488049128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389277778242824 % _timestamp‚1.7845009793166943e+09 _runtime‚4053.227822181 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389277675054878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76436974 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387250738415616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7067480723968 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394318202100224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382232290193696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101861309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25808696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7045401900542 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101871466 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101871466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚377167187527168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7273358707200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44249934 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7067480723968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178308442 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178308442 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚700586303fvì&çæM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76436976 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70058866 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387250738415616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚382232273154336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚814 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7045488049128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389277778242824 % _timestamp‚1.7845009793166943e+09 _runtime‚4053.227822181 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389277675054878 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76436974 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387250738415616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7067480723968 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394318202100224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382232290193696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101861309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25808696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7045401900542 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101871466 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101871466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚377167187527168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7273358707200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44249934 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7067480723968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178308442 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178308442 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70058630 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387250721376256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303®U~R2è- _wandbruntime‚4053  _runtime‚4053ùd¶H é† 2loss/SRqqt2/assignment_loss‚0.5706016528141239 3loss/FBt2/assignment_loss‚0.0013830007593745022 /loss/FRt2/detection_loss‚1.3383291236063803 1loss/SRqqt1/detection_loss‚0.8908810453214402 1loss/SRqqt2/detection_loss‚0.5913637768614232 $loss/total_loss‚8.86578146738314 trainer/global_step‚10049 2loss/FBt1/assignment_loss‚0.030444438317857703 /loss/FBt1/detection_loss‚0.8029303734043711 0loss/FRt1/assignment_loss‚2.0815750391212937 /loss/FRt1/detection_loss‚0.4806335542136479 epoch‚12 % _timestamp‚1.7845009793176208e+09 0loss/FRt2/assignment_loss‚0.8285439035146853 2loss/SRqqt1/assignment_loss‚0.8255545914027802 0loss/FBt2/detection_loss‚0.42354096804576163 _runtime‚4053.228361302 _step‚815­Âééê‹ 1loss/SRqqt2/detection_loss‚0.5913637768614232 $loss/total_loss‚8.86578146738314 trainer/global_step‚10049 2loss/FBt1/assignment_loss‚0.030444438317857703 /loss/FBt1/detection_loss‚0.8029303734043711 0loss/FRt1/assignment_loss‚2.0815750391212937 /loss/FRt1/detection_loss‚0.4806335542136479 epoch‚12 % _timestamp‚1.7845009793176208e+09 0loss/FRt2/assignment_loss‚0.8285439035146853 2loss/SRqqt1/assignment_loss‚0.8255545914027802 0loss/FBt2/detection_loss‚0.42354096804576163 _runtime‚4053.228361302 _step‚815 2loss/SRqqt2/assignment_loss‚0.5706016528141239 3loss/FBt2/assignment_loss‚0.0013830007593745022 /loss/FRt2/detection_loss‚1.3383291236063803 1loss/SRqqt1/detection_loss‚0.8908810453214402¯ {j¸oë‚(:ä ÿ¥õÒÄÐÌ+ disk.md0.out‚ 120.2421875 proc.cpu.threads‚17 network.sent‚ 1.3401098e+07 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚51592.00146484375 disk.nvme0n1.in‚ 16.43359375 disk.nvme1n1.in‚3.25 disk.md0.in‚0 network.recv‚ 1.191599e+06% disk./.usageGB‚1167.6696853637695" disk.dm-1.out‚2971.76318359375 disk.rbd0.in‚0& proc.memory.availableMB‚ 133.546875* proc.memory.percent‚17.381906509399414 cpu‚26.296734653557284! disk.nvme0c0n1.in‚ 16.43359375& disk.nvme0n1.out‚51592.00146484375$ memory_percent‚99.18489456176758" proc.memory.rssMB‚ 2847.8515625) disk./.usagePercent‚76.05729916064287# disk.dm-0.out‚73056.34228515625 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚24692.79736328125& disk.nvme1n1.out‚24692.79736328125š›ì‚(: ÿ¥õÒŽ°©, gpu.0.gpu‚67.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚174.098' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚58.032666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ü'[2í- _wandbruntime‚4073  _runtime‚4073O$ÑÇîÁ & lr-AdamW/pg1‚8.830537538792734e-05 & lr-AdamW/pg2‚8.830537538792734e-05 trainer/global_step‚10099 % _timestamp‚1.7845009992359502e+09 _runtime‚ 4073.14775654 _step‚816*¿ÌïÆ trainer/global_step‚10099 % _timestamp‚1.7845009992359502e+09 _runtime‚ 4073.14775654 _step‚816 & lr-AdamW/pg1‚8.830537538792734e-05 & lr-AdamW/pg2‚8.830537538792734e-05°ÉÔÔ½2ð- _wandbruntime‚4073  _runtime‚4073ðŸ»Û'ñÕO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102358306 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76795705 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7100740356096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44460144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389081690409984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179154011 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396182413726720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102348163 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7100828378112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371642702730240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10099 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845009992367897e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102358306 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚396182518788096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396182518788096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389081673370624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44460147 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378950459290112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102348163 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70390163 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76795703 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7307756559872 _step‚817 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389081690409984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76795705 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378950247681536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384039021513936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25930268 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7078641572980 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76795703 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25930019 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _runtime‚ 4073.1478211 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179143866 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396182413726720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44460147 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378950459290112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102348163 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70390163 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76795703 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7307756559872 _step‚817 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389081690409984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76795705 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378950247681536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384039021513936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25930268 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7078641572980 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76795703 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25930019 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _runtime‚ 4073.1478211 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179143866 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396182413726720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102358306 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76795705 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7100740356096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44460144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389081690409984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179154011 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396182413726720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102348163 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7100828378112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371642702730240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10099 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845009992367897e+09±½£M÷2ó- _wandbruntime‚4073  _runtime‚4073¶Ý¹£æ&ôàM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389119039737856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70396791 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102358061 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102358061 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚10099 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179171244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102368218 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70397062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179171244 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179161085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚4073.422425923 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44464437 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚391155107918286 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384075876542336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7079231375950 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371678372983808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76803024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76803024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44464434 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚396220563272192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389119039737856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7101506494976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7079317524536 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7101418465792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389119056777216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7101506494976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76803026 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚396220458203648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7101418465792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179161085 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384075893581696K3ã1ë&õåM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384075876542336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7079231375950 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371678372983808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76803024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76803024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44464434 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚396220563272192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389119039737856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7101506494976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7079317524536 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7101418465792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389119056777216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7101506494976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76803026 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚396220458203648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7101418465792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179161085 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384075893581696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389119039737856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70396791 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102358061 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102358061 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚10099 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179171244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102368218 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70397062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179171244 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179161085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚4073.422425923 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44464437 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚391155107918286 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464²!:æ(2ö- _wandbruntime‚4073  _runtime‚4073xÒ+Œ ÷… 0loss/FRt2/assignment_loss‚0.9686935339856335 /loss/FRt1/detection_loss‚0.5873691850546069 0loss/SRqqt1/detection_loss‚1.175118187010683 1loss/SRqqt2/detection_loss‚0.6780309495627618 2loss/SRqqt1/assignment_loss‚0.6856328591022838 3loss/FBt2/assignment_loss‚0.0012739869242157351 /loss/FBt2/detection_loss‚0.4855007558953071 %loss/total_loss‚9.323034502209008 _runtime‚4073.423080815 _step‚819 epoch‚12 0loss/FRt1/assignment_loss‚2.1295421575677804 2loss/SRqqt2/assignment_loss‚0.5941511673537988 2loss/FBt1/assignment_loss‚0.023807722097002326 /loss/FRt2/detection_loss‚1.4966214223449508 trainer/global_step‚10099 % _timestamp‚1.7845009995124154e+09 /loss/FBt1/detection_loss‚0.4972925753099844¯¢üøŠ /loss/FBt1/detection_loss‚0.4972925753099844 0loss/FRt2/assignment_loss‚0.9686935339856335 /loss/FRt1/detection_loss‚0.5873691850546069 0loss/SRqqt1/detection_loss‚1.175118187010683 1loss/SRqqt2/detection_loss‚0.6780309495627618 2loss/SRqqt1/assignment_loss‚0.6856328591022838 3loss/FBt2/assignment_loss‚0.0012739869242157351 /loss/FBt2/detection_loss‚0.4855007558953071 %loss/total_loss‚9.323034502209008 _runtime‚4073.423080815 _step‚819 epoch‚12 0loss/FRt1/assignment_loss‚2.1295421575677804 2loss/SRqqt2/assignment_loss‚0.5941511673537988 2loss/FBt1/assignment_loss‚0.023807722097002326 /loss/FRt2/detection_loss‚1.4966214223449508 trainer/global_step‚10099 % _timestamp‚1.7845009995124154e+09³|}púiù‚(:Þ Ž¦õÒÑ­÷+ disk.dm-1.in‚0 disk.nvme0n1.in‚ 16.43359375' disk.nvme1c1n1.out‚24693.4208984375 disk.md0.out‚ 120.6328125* proc.memory.percent‚17.381906509399414% disk./.usageGB‚1167.6731033325195 disk.dm-0.in‚ 19.68359375! disk.dm-1.out‚2971.8935546875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375' disk.nvme0c0n1.out‚52569.6025390625% disk.nvme1n1.out‚24693.4208984375( proc.memory.availableMB‚ 133.65234375) disk./.usagePercent‚76.05752179335741 disk.rbd0.in‚0% disk.nvme0n1.out‚52569.6025390625 disk.nvme1c1n1.in‚3.25 disk.md0.in‚0" proc.memory.rssMB‚ 2847.8515625 cpu‚26.299857008518394 proc.cpu.threads‚17 network.sent‚ 1.3477387e+07" disk.dm-0.out‚74033.6064453125 disk.nvme1n1.in‚3.25" memory_percent‚99.184250831604 network.recv‚ 1.195625e+06Û0ún™ú‚(:Ž Ž¦õÒ½ë£, gpu.0.gpu‚76.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚132.49' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚44.16333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïæ2û- _wandbruntime‚4093  _runtime‚4093©)]²Èü & lr-AdamW/pg2‚8.816713954686336e-05 trainer/global_step‚10149 % _timestamp‚1.7845010197444181e+09 _runtime‚4093.655352617 _step‚820 & lr-AdamW/pg1‚8.816713954686336e-05éôðÍýÇ % _timestamp‚1.7845010197444181e+09 _runtime‚4093.655352617 _step‚820 & lr-AdamW/pg1‚8.816713954686336e-05 & lr-AdamW/pg2‚8.816713954686336e-05 trainer/global_step‚10149´vMÄ2þ- _wandbruntime‚4093  _runtime‚4093õÁËb‚ÿ×O TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26052864 aMDeviceStatsMonitor.on_train_batch_wÊì¨['start/inactive_split_bytes.large_pool.freed‚373427435688960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390949991732224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102855058 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70727508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚212 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385882607862576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7134766119936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70727723 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398084669830144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385882624901936 % _timestamp‚1.7845010197453847e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚398084669830144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390950008771584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380769980329472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373427460592640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚10149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚215 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7342712471040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102844915 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚4093.659432594 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392994992762490 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77161753 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77161755 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102844915 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390949991732224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180016813 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398084774891520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390950008771584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392995095950324 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7112384899914 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102855058 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380770173063680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77161753 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77161755 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102844915 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390949991732224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180016813 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398084774891520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390950008771584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392995095950324 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7112384899914 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102855058 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380770173063680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77161753 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102855058 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70727508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚212 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385882607862576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7134766119936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70727723 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398084669830144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385882624901936 % _timestamp‚1.7845010197453847e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚398084669830144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390950008771584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380769980329472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373427460592640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚10149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚215 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7342712471040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102844915 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚4093.659432594 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392994992762490µE²2- _wandbruntime‚4093  _runtime‚4093̵ÚÅç&‚áM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7113060851358 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77169076 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390987358099456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393032540781694 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7135356207616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102864970 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7343425544192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393032643969640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390987375138816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77169076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385919479930336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102864970 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180023887 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398122819375616 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180034046 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7343232555008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77169074 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚390987358099456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380806581294080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102854813 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70734466 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚398122714307072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7135356207616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26055292 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7113146999944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102854813 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380806363401216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180034046 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7135444236800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385919496969696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44678937 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚822 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26055529 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70734226 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77169074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 % _timestamp‚1.7845010200193822e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180023887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚10149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398122819375616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390987375138816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7135444236800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374À%#‡ ƒæM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77169076 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390987358099456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393032540781694 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7135356207616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102864970 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7343425544192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393032643969640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390987375138816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77169076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385919479930336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102864970 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180023887 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398122819375616 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180034046 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7343232555008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77169074 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚390987358099456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380806581294080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102854813 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70734466 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚398122714307072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7135356207616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26055292 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7113146999944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102854813 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380806363401216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180034046 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7135444236800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385919496969696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44678937 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚822 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26055529 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70734226 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77169074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 % _timestamp‚1.7845010200193822e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180023887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚10149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398122819375616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390987375138816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7135444236800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7113060851358¶Ü£m2„- _wandbruntime‚4093  _runtime‚4093´ä‘ …ƒ 1loss/SRqqt2/detection_loss‚0.6941483769799967 %loss/total_loss‚9.321190946409825 0loss/FRt1/assignment_loss‚2.0706577268841455 /loss/FBt2/detection_loss‚0.4516291192865797 trainer/global_step‚10149 % _timestamp‚1.7845010200204322e+09 _runtime‚4093.931294365 _step‚823 0loss/FRt2/assignment_loss‚0.9991668509550331 .loss/FRt2/detection_loss‚1.423692762089724 1loss/SRqqt1/detection_loss‚1.0347803457356666 -loss/FBt1/detection_loss‚0.75610566136482 2loss/SRqqt1/assignment_loss‚0.6297702555267556 2loss/SRqqt2/assignment_loss‚0.6373568865930412 2loss/FBt1/assignment_loss‚0.002393555253940563 3loss/FBt2/assignment_loss‚0.0016098641261351077 epoch‚12 /loss/FRt1/detection_loss‚0.6198795416139881bgz †ˆ -loss/FBt1/detection_loss‚0.75610566136482 2loss/SRqqt1/assignment_loss‚0.6297702555267556 2loss/SRqqt2/assignment_loss‚0.6373568865930412 2loss/FBt1/assignment_loss‚0.002393555253940563 3loss/FBt2/assignment_loss‚0.0016098641261351077 epoch‚12 /loss/FRt1/detection_loss‚0.6198795416139881 1loss/SRqqt2/detection_loss‚0.6941483769799967 %loss/total_loss‚9.321190946409825 0loss/FRt1/assignment_loss‚2.0706577268841455 /loss/FBt2/detection_loss‚0.4516291192865797 trainer/global_step‚10149 % _timestamp‚1.7845010200204322e+09 _runtime‚4093.931294365 _step‚823 0loss/FRt2/assignment_loss‚0.9991668509550331 .loss/FRt2/detection_loss‚1.423692762089724 1loss/SRqqt1/detection_loss‚1.0347803457356666·†lík‡‚(:à ¦õÒÒßÓ+ proc.cpu.threads‚17) disk./.usagePercent‚76.05752427809752! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 disk.md0.out‚ 121.27734375! disk.dm-1.out‚2971.9677734375 network.sent‚ 1.3538534e+07% disk.nvme1c1n1.out‚24696.17578125 disk.md0.in‚0( proc.memory.availableMB‚ 133.35546875 cpu‚26.302295495851872 network.recv‚ 1.199919e+06 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚52580.69287109375 disk.nvme0n1.in‚ 16.43359375& disk.nvme0n1.out‚52580.69287109375 disk.nvme1n1.in‚3.25# disk.dm-0.out‚74046.03955078125% disk./.usageGB‚1167.6731414794922# disk.nvme1n1.out‚24696.17578125 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0$ memory_percent‚99.18606281280518" proc.memory.rssMB‚ 2847.8515625* proc.memory.percent‚17.381906509399414Û‘÷àšˆ‚(: ¦õÒ¹´û+ gpu.0.gpu‚70.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚151.291' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚50.43033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ>·Ž‰‚(:ƒ ¬¦õÒœöœ+ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚196.392' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û’lŠ‚(:á ¬¦õÒÆá¯+ network.sent‚ 1.3587926e+07 network.recv‚ 1.203918e+06) disk./.usagePercent‚76.05730487554513 disk.nvme1n1.in‚3.25 disk.md0.out‚ 121.93359375 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚3.25" disk.dm-1.out‚2972.17724609375' disk.nvme0c0n1.out‚52593.8212890625 disk.nvme0n1.in‚ 16.43359375% disk.nvme0n1.out‚52593.8212890625$ memory_percent‚99.19004440307617" proc.memory.rssMB‚ 2847.8515625 cpu‚26.300653785240232% disk./.usageGB‚1167.6697731018066% disk.nvme1n1.out‚24698.1845703125! disk.nvme0c0n1.in‚ 16.43359375& proc.memory.availableMB‚ 132.703125* proc.memory.percent‚17.381906509399414' disk.nvme1c1n1.out‚24698.1845703125 disk.md0.in‚0# disk.dm-0.out‚74059.57177734375 disk.rbd0.out‚0 proc.cpu.threads‚17ñýï-2‹- _wandbruntime‚4113  _runtime‚4113û®šÇŒÁ _runtime‚4113.945833414 _step‚824 & lr-AdamW/pg1‚8.802830645597845e-05 & lr-AdamW/pg2‚8.802830645597845e-05 trainer/global_step‚10199 $ _timestamp‚1.784501040034854e+09öÐÐûÌÆ & lr-AdamW/pg1‚8.802830645597845e-05 & lr-AdamW/pg2‚8.802830645597845e-05 trainer/global_step‚10199 $ _timestamp‚1.784501040034854e+09 _runtime‚4113.945833414 _step‚824¸mdDò2Ž- _wandbruntime‚4113  _runtime‚4113ûî.(Ý'×O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387726211250576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77527803 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7168615839744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4113.950416143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26176763 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚375212193551360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7377498279424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7168615839744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77527805 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7168703861760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44889147 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394872425625898 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399987030994944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392818327133184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44889144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26176514 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394872528813732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382589897147904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7377678692864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7168703861760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382589691830784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7146300523796 % _timestamp‚1.7845010400359306e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71065658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71065910 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392818310093824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392818310093824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399986925933568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 trainer/global_step‚10199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180869470 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392818327133184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚825 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚375212218455040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399986925933568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77527803 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180879615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103341667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399987030994944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527805 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238ƒ[Áâ'ÜO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103351810 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103351810 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103341667 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392818327133184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44889144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26176514 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394872528813732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382589897147904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7377678692864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7168703861760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382589691830784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7146300523796 % _timestamp‚1.7845010400359306e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71065658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71065910 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392818310093824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392818310093824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399986925933568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 trainer/global_step‚10199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180869470 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392818327133184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚825 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚375212218455040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399986925933568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77527803 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180879615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103341667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399987030994944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527805 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387726211250576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77527803 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7168615839744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4113.950416143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26176763 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚375212193551360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7377498279424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7168615839744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77527805 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7168703861760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44889147 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394872425625898 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399987030994944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚375247913612288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400025075479040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392855693500416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 trainer/global_step‚10199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180886689 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7169381978624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71072377 % _timestamp‚1.7845010403097844e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400025075479040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26178943 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103351565 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7378190357504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180896848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103361722 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7169293949440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103361722 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77535126 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77535124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387763100357696 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7146976475352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392855676461056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7169293949440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚375247888708608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103351565 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚826 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71072624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392855693500416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180896848 _runtime‚4114.220901293 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚375247913612288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400025075479040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392855693500416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 trainer/global_step‚10199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180886689 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7169381978624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71072377 % _timestamp‚1.7845010403097844e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400025075479040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26178943 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103351565 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7378190357504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180896848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103361722 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7169293949440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103361722 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77535126 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77535124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387763100357696 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7146976475352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392855676461056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7169293949440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚375247888708608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103351565 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚826 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71072624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928ºY¾ýq2”- _wandbruntime‚4114  _runtime‚4114—ºÐ^ •„ 0loss/FRt2/assignment_loss‚0.9447401555116511 2loss/FBt2/assignment_loss‚0.001254146131929033 /loss/FRt1/detection_loss‚0.5877010591804465 0loss/FBt1/detection_loss‚0.38219701717405213 %loss/total_loss‚9.402856953678741 2loss/SRqqt2/assignment_loss‚0.5752261214346861 1loss/SRqqt1/detection_loss‚1.0001079670072301 /loss/FBt2/detection_loss‚0.7176883753441695 _step‚827 0loss/FRt1/assignment_loss‚2.2060745787468687 /loss/FRt2/detection_loss‚1.4832438371944712 1loss/SRqqt2/detection_loss‚0.6061471410679408 trainer/global_step‚10199 _runtime‚4114.225381851 2loss/SRqqt1/assignment_loss‚0.8015147341577799 1loss/FBt1/assignment_loss‚0.09696182072751623 epoch‚12 $ _timestamp‚1.784501040310833e+09P™7Ø–‰ %loss/total_loss‚9.402856953678741 2loss/SRqqt2/assignment_loss‚0.5752261214346861 1loss/SRqqt1/detection_loss‚1.0001079670072301 /loss/FBt2/detection_loss‚0.7176883753441695 _step‚827 0loss/FRt1/assignment_loss‚2.2060745787468687 /loss/FRt2/detection_loss‚1.4832438371944712 1loss/SRqqt2/detection_loss‚0.6061471410679408 trainer/global_step‚10199 _runtime‚4114.225381851 2loss/SRqqt1/assignment_loss‚0.8015147341577799 1loss/FBt1/assignment_loss‚0.09696182072751623 epoch‚12 $ _timestamp‚1.784501040310833e+09 0loss/FRt2/assignment_loss‚0.9447401555116511 2loss/FBt2/assignment_loss‚0.001254146131929033 /loss/FRt1/detection_loss‚0.5877010591804465 0loss/FBt1/detection_loss‚0.38219701717405213»b0 ío—‚(:ä »¦õÒÜ‚­+ network.sent‚ 1.3592786e+07 network.recv‚ 1.207523e+06& disk.nvme0n1.out‚52604.54541015625& disk.nvme1n1.out‚24699.35009765625 disk.dm-1.in‚0$ memory_percent‚99.18885231018066" proc.memory.rssMB‚ 2847.8515625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚52604.54541015625( disk.nvme1c1n1.out‚24699.35009765625# disk.dm-0.out‚74070.54443359375* proc.memory.percent‚17.381906509399414( disk./.usagePercent‚75.8317086114652% disk./.usageGB‚1164.2063064575195 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.dm-0.in‚ 19.68359375" disk.dm-1.out‚2972.25537109375 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme0n1.in‚ 16.43359375 disk.md0.in‚0 disk.md0.out‚ 122.31640625' proc.memory.availableMB‚ 132.8984375 cpu‚26.29914136937151Vt4€š˜‚(: »¦õÒ¡ºÝ+ gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚197.753' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.91766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$)CÙ2™- _wandbruntime‚4134  _runtime‚4134ÁQ.ÇšÁ & lr-AdamW/pg1‚8.788887833896632e-05 & lr-AdamW/pg2‚8.788887833896632e-05 trainer/global_step‚10249 $ _timestamp‚1.784501060129786e+09 _runtime‚4134.042042538 _step‚828.ÒêôÌ›Æ trainer/global_step‚10249 $ _timestamp‚1.784501060129786e+09 _runtime‚4134.042042538 _step‚828 & lr-AdamW/pg1‚8.788887833896632e-05 & lr-AdamW/pg2‚8.788887833896632e-05¼¸°è2œ- _wandbruntime‚4134  _runtime‚4134™½í&Ž$ÖO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384409395111424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376996976317440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26299823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376996951413760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103848562 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401889182036992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚394686628455424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384409600428544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚401889287098368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394686628455424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103848562 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7180043850730 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7202553581568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45103644 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71403470 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181732272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚829 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103838419 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77893853 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7412624111104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7412443697664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77893853 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103838419 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45103647 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394686645494784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181742417 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26299577 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7202553581568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7180129999204 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Ïžiá'žÛO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394686645494784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚4134.042773469 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389569831677936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396749961677140 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103838419 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77893853 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7412624111104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7412443697664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77893853 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103838419 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45103647 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394686645494784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181742417 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26299577 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7202553581568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7180129999204 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384409395111424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376996976317440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26299823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376996951413760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103848562 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401889182036992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚394686628455424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384409600428544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚401889287098368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394686628455424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103848562 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7180043850730 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7202553581568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45103644 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71403470 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181732272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚829 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394723994822656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4134.299043759 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401927331582464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181749491 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77901176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71409928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103848317 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚830 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26301994 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7180719802174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377032671474688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚389606686706336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7203231691264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384445992070144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396787509696456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401927226513920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77901174 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26302232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71410169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181759650 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚396787406508510 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181759650 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77901174 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152¥?<ë&¡åM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7203231691264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384445992070144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396787509696456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401927226513920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77901174 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26302232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71410169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181759650 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚396787406508510 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181759650 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77901174 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389606703745696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7203319720448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7203231691264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384445782565888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394723994822656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4134.299043759 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401927331582464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181749491 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77901176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71409928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103848317 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚830 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26301994 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7180719802174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377032671474688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚389606686706336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808¾ˆQŒ@2¢- _wandbruntime‚4134  _runtime‚4134¨,ߣ‚ 1loss/SRqqt2/assignment_loss‚0.640029661935862 1loss/FBt1/assignment_loss‚0.18476304120863232 /loss/FRt1/detection_loss‚0.5949090733838757 trainer/global_step‚10249 % _timestamp‚1.7845010603888938e+09 3loss/FBt2/assignment_loss‚0.0073515592731821355 /loss/FRt2/detection_loss‚1.3533173722865166 %loss/total_loss‚9.693854209687737 /loss/FRt1/assignment_loss‚2.144428467599012 1loss/SRqqt1/detection_loss‚1.0977984640594054 /loss/FBt1/detection_loss‚0.4750045061856508 0loss/FRt2/assignment_loss‚1.0351943409667825 0loss/SRqqt2/detection_loss‚0.669786132223599 /loss/FBt2/detection_loss‚0.5963938242266289 epoch‚12 _runtime‚4134.299506199 _step‚831 2loss/SRqqt1/assignment_loss‚0.8948777663385896PÓÝo ¤‡ /loss/FRt1/assignment_loss‚2.144428467599012 1loss/SRqqt1/detection_loss‚1.0977984640594054 /loss/FBt1/detection_loss‚0.4750045061856508 0loss/FRt2/assignment_loss‚1.0351943409667825 0loss/SRqqt2/detection_loss‚0.669786132223599 /loss/FBt2/detection_loss‚0.5963938242266289 epoch‚12 _runtime‚4134.299506199 _step‚831 2loss/SRqqt1/assignment_loss‚0.8948777663385896 1loss/SRqqt2/assignment_loss‚0.640029661935862 1loss/FBt1/assignment_loss‚0.18476304120863232 /loss/FRt1/detection_loss‚0.5949090733838757 trainer/global_step‚10249 % _timestamp‚1.7845010603888938e+09 3loss/FBt2/assignment_loss‚0.0073515592731821355 /loss/FRt2/detection_loss‚1.3533173722865166 %loss/total_loss‚9.693854209687737¿r+MÄH¥‚(:ã ʦõÒÓÚî+ network.sent‚ 1.3667951e+07% disk.nvme1n1ß_ÝÂ&.out‚24701.9189453125 disk.md0.in‚0# disk.dm-0.out‚78376.85595703125* proc.memory.percent‚17.381906509399414% disk./.usageGB‚1169.5398445129395 disk.nvme1n1.in‚3.25! disk.dm-1.out‚2972.3896484375 disk.nvme0n1.in‚ 16.43359375( proc.memory.availableMB‚ 132.83984375) disk./.usagePercent‚76.17911379338476 disk.nvme1c1n1.in‚3.25' disk.nvme1c1n1.out‚24701.9189453125 disk.md0.out‚ 122.59765625 disk.rbd0.in‚0& disk.nvme0n1.out‚56909.04150390625 cpu‚26.30016097431951 network.recv‚ 1.211812e+06 disk.dm-0.in‚ 19.68359375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375( disk.nvme0c0n1.out‚56909.04150390625$ memory_percent‚99.18920993804932" proc.memory.rssMB‚ 2847.8515625 proc.cpu.threads‚17 disk.dm-1.in‚0d˼™¦‚(:Ž ʦõÒ›¢, gpu.0.gpu‚76.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚198.56' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.18666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dY`@2§- _wandbruntime‚4153  _runtime‚4153Ê:MéȨ % _timestamp‚1.7845010800426948e+09 _runtime‚4153.953698205 _step‚832 & lr-AdamW/pg1‚8.774885742905129e-05 & lr-AdamW/pg2‚8.774885742905129e-05 trainer/global_step‚10299màŠÍ©Ç trainer/global_step‚10299 % _timestamp‚1.7845010800426948e+09 _runtime‚4153.953698205 _step‚832 & lr-AdamW/pg1‚8.774885742905129e-05 & lr-AdamW/pg2‚8.774885742905129e-05Àæ­,>2ª- _wandbruntime‚4153  _runtime‚4153âÑG•Ý'«×O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚391413435065936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71740594 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182605219 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386229295514624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7236491323392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78259905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26422193 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378781734179840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378781709276160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396554963856384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182595074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26422447 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396554946817024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78259905 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚10299 _step‚833 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7236579345408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403791438140416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78259903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚396554963856384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71740337 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398627394540548 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7447372532736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398627291352714 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386229081808896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚391413418026576 _runtime‚4153.954779627 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78259903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7236491323392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45318144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104335171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845010800438263e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104345314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7213959474612 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403791438140416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104345314 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45318147 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7236579345408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403791438140416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78259903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚396554963856384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71740337 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398627394540548 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7447372532736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398627291352714 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386229081808896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚391413418026576 _runtime‚4153.954779627 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78259903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7236491323392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45318144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104335171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845010800438263e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104345314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7213959474612 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403791438140416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104345314 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45318147 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386229295514624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7236491323392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78259905 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26422193 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378781734179840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378781709276160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396554963856384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182595074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26422447 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816Á-_f[2­- _wandbruntime‚4154  _runtime‚4154‡ÝB.ç&®áM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71747017 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104355226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403829587685888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398664942559864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78267226 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45322434 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26424821 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398664839371918 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104345069 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104355226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104345069 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45322437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚396592313184256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403829482617344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396592330223616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7237257462272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391450307133696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78267224 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7214635426168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403829587685888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182622452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚4154.229535978 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182622452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378817404433408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78267224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378817429337088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391450307133696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78267224 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7214635426168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403829587685888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182622452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚4154.229535978 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182622452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378817404433408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78267224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378817429337088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71747017 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104355226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403829587685888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398664942559864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78267226 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45322434 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26424821 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398664839371918 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104345069 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104355226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104345069 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45322437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚396592313184256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403829482617344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396592330223616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7237257462272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405693799305216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405693799305216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398423282217984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380566467138560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845010998998034e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78625955 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7247788950020 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚400504827403956 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _runtime‚4173.812346398 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72078127 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7247702801546 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚837 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78625955 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7270517087232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚398423265178624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104842066 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183457876 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚10349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400504724216122 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7270429065216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405693694243840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7270429065216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380566492042240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398423265178624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78625953 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104842066 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45532644 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0I½öe¯ ¼ÜO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45532644 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104831923 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405693799305216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405693799305216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398423282217984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380566467138560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845010998998034e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78625955 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7247788950020 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚400504827403956 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _runtime‚4173.812346398 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72078127 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7247702801546 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚837 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78625955 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7270517087232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚398423265178624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104842066 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183457876 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚10349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400504724216122 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7270429065216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405693694243840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7270429065216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380566492042240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398423265178624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78625953 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104842066 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143ůà+2½- _wandbruntime‚4174  _runtime‚4174ˆVtÖç&¾áM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183485254 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104851978 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45536937 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7271107174912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78633274 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104841821 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7248378752990 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398460648585216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7271107174912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7483018763776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45536934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7248464901576 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72084654 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26547720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78633276 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104851978 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7271195204096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380602187199488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400542375423272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398460631545856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388085392660992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393293893482336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845011004995015e+09 _runtime‚4174.410684571 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78633276 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183485254 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78633274 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26547959 trainer/global_step‚10349 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248X[ú3ì&¿æM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183475095 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398460648585216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7483205461504 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚405731843789312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380602162295808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183475095 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7271107174912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78633274 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104841821 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7248378752990 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398460648585216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7271107174912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7483018763776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45536934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7248464901576 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72084654 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26547720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78633276 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104851978 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7271195204096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380602187199488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400542375423272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398460631545856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388085392660992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393293893482336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845011004995015e+09 _runtime‚4174.410684571 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78633276 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183485254 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78633274 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26547959 trainer/global_step‚10349 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183485254 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104851978 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45536937 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184320678 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚10399 % _timestamp‚1.7845011201943557e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚841 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78992003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407595950347264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78992005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402382157079530 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26668912 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389868699037184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7281618425428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407596055408640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105328675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26669158 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382351249904640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78992005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7281532276954 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184330823 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78992003 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400291600579584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105328675 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7304366807040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7517449132544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72416305 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceS‡Ž³ä¦ tatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382351225000960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7517264524800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72416056 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400291583540224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0™×"[â'ÊÜO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400291600579584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395100624802576 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105338818 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _runtime‚4194.105267542 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45747144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7304366807040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400291583540224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45747147 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400291583540224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105338818 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184320678 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚10399 % _timestamp‚1.7845011201943557e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _step‚841 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78992003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407595950347264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78992005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402382157079530 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26668912 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389868699037184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7281618425428 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407596055408640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105328675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26669158 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382351249904640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78992005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7281532276954 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184330823 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78992003 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400291600579584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105328675 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7304366807040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7517449132544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72416305 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382351225000960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7517264524800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72416056 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72422828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395137496870336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72423069 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45751434 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184337897 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78999324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184348056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382386945061888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7517962810368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚842 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400328966946816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105348730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26671632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7305044916736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78999324 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚389905094569984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105348730 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402419705098734 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184348056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78999326 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400328949907456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7282208228398 $ _timestamp‚1.784501120469339e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389904882968576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105338573 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7305132945920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7305044916736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105338573 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407633994824192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157ÚÅæ-=&ÍåM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400328966946816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105348730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26671632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7305044916736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78999324 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚389905094569984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105348730 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402419705098734 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184348056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78999326 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400328949907456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7282208228398 $ _timestamp‚1.784501120469339e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389904882968576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105338573 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7305132945920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7305044916736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105338573 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407633994824192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72422828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395137496870336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72423069 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45751434 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184337897 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78999324 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184348056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382386945061888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7517962810368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 QCDeviceStatsÂÀHf®Monitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚842ʯ#o2Î- _wandbruntime‚4194  _runtime‚4194¯ç%x Ï„ /loss/FRt1/detection_loss‚0.5682417393122305 /loss/FRt2/detection_loss‚1.3743649711167174 /loss/FBt2/detection_loss‚0.8153911587998163 $ _timestamp‚1.784501120470385e+09 _runtime‚4194.380997967 0loss/FRt2/assignment_loss‚0.7767715673361506 2loss/SRqqt2/assignment_loss‚0.5278498156452985 1loss/SRqqt2/detection_loss‚0.6165212200267903 0loss/FBt1/detection_loss‚0.43410615447296697 2loss/FBt2/assignment_loss‚0.006169969182113937 1loss/SRqqt1/detection_loss‚0.9745527456054487 %loss/total_loss‚8.968022439056753 /loss/FRt1/assignment_loss‚2.044732799437392 2loss/SRqqt1/assignment_loss‚0.8264427826717407 epoch‚12 trainer/global_step‚10399 _step‚843 2loss/FBt1/assignment_loss‚0.002877515450087742Pß&¾Ð‰ 2loss/SRqqt2/assignment_loss‚0.5278498156452985 1loss/SRqqt2/detection_loss‚0.6165212200267903 0loss/FBt1/detection_loss‚0.43410615447296697 2loss/FBt2/assignment_loss‚0.006169969182113937 1loss/SRqqt1/detection_loss‚0.9745527456054487 %loss/total_loss‚8.968022439056753 /loss/FRt1/assignment_loss‚2.044732799437392 2loss/SRqqt1/assignment_loss‚0.8264427826717407 epoch‚12 trainer/global_step‚10399 _step‚843 2loss/FBt1/assignment_loss‚0.002877515450087742 /loss/FRt1/detection_loss‚0.5682417393122305 /loss/FRt2/detection_loss‚1.3743649711167174 /loss/FBt2/detection_loss‚0.8153911587998163 $ _timestamp‚1.784501120470385e+09 _runtime‚4194.380997967 0loss/FRt2/assignment_loss‚0.7767715673361506Ë]9;3›Ñ‚(: †§õÒ¨¨“- gpu.0.gpu‚44.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.683' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.894333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©9ÂâhÒ‚(:Ý †§õÒË׌/( proc.memory.availableMB‚ 132.73046875( disk.nvme0c0n1.out‚58876.99462890625 disk.md0.in‚0* proc.memory.percent‚17.381906509399414% disk.nvme1c1n1.out‚24926.08203125# disk.nvme1n1.out‚24926.08203125 network.recv‚ 1.227511e+06( disk./.usagePercent‚76.1795983177077 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 disk.md0.out‚ 124.2265625 disk.nvme0n1.in‚ 16.43359375& disk.nvme0n1.out‚58876.99462890625 disk.nvme1n1.in‚3.25# disk.dm-0.out‚80565.06005859375# memory_percent‚99.1898775100708" proc.memory.rssMB‚ 2847.8515625 cpu‚26.295746510750625 proc.cpu.threads‚17 network.sent‚ 1.3860754e+07% disk./.usageGB‚1169.5472831726074 disk.rbd0.in‚0 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0! disk.dm-1.out‚2972.8271484375á"…mšÓ‚(: •§õÒ¼÷Í+ gpu.0.gpu‚53.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚198.484' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.16133333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚003;gÔ‚(:Ü •§õÒÞÂæ+ network.sent‚ 1.3910941e+07) disk./.usagePercent‚76.23447032456987& disk.nvme1c1n1.out‚24932.490234375! disk.dm-1.out‚2972.9208984375* proc.memory.percent‚17.442822456359863 cpu‚26.30542135158799 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375$ disk.nvme1n1.out‚24932.490234375 network.recv‚ 1.232034e+06( disk.nvme0c0n1.out‚62291.54443359375 disk.nvme0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25' proc.memory.availableMB‚ 122.5859375# proc.memory.rssMB‚ 2857.83203125$ disk./.usageGB‚1170.389705657959& disk.nvme0n1.out‚62291.54443359375 disk.md0.in‚0 disk.md0.out‚124.8125 disk.dm-0.in‚ 19.68359375# disk.dm-0.out‚83984.67822265625 disk.dm-1.in‚0$ memory_percent‚99.25179481506348Äî7î2Õ- _wandbruntime‚4216  _runtime‚4216lUâvj=Öäz <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5169271230697632 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8577901124954224 )loss/val_total_loss‚8.930482542674769 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18512195348739624 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9248929023742676 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3898107409477234 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7validation_average_jet_accuracy‚0.36881721019744873 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3672839403152466 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984210729598999 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6995879411697388 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5549590587615967 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2036711871623993 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 +jet/accuracy_3_of_3‚0.07163427770137787 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17184458673000336 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992790222167969 particle/accuracy_3_of_5‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18185269832611084 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46076062321662903 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04128874093294144 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5503841638565063 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7664803266525269 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8455668091773987 *jet/accuracy_2_of_4‚0.8004511594772339 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6972751021385193 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1728065460920334 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20625947415828705 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16384775936603546 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04128874093294144 +jet/accuracy_3_of_4‚0.09097576141357422 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _runtime‚4216.054624555 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.6622011661529541 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959508776664734 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18551819026470184 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22436943650245667 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 epoch‚12 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02447054721415043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8027777671813965 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9977248907089233 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998987078666687 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14644339680671692 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 +jet/accuracy_2_of_2‚0.15388260781764984 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6573659181594849 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6880514025688171 *jet/accuracy_1_of_2‚0.8004854917526245 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15792810916900635 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2881862223148346 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04650837555527687 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04718945547938347 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7724941968917847 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 $ _timestamp‚1.784501142142975e+09 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1908876597881317 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6924315690994263 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7477802038192749 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.018498167395591736 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20897305011749268 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.046843983232975006 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.021527601405978203 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7974358797073364 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7894736528396606 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940045475959778 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7436095476150513 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13790000975131989 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19338984787464142 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8455668091773987 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9148052930831909 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18362383544445038 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21298234164714813 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6845167875289917 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.047217678278684616 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46637845039367676 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6416815519332886 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5685773491859436 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012398983351886272 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5761831998825073 particle/accuracy_2_of_3‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865512847900391 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15248605608940125 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03625745698809624 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1713656187057495 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7894736528396606 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04771891608834267 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3222823143005371 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8070175051689148 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3921290934085846 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14841283857822418 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959927797317505 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 trainer/global_step‚10412 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7264717817306519 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7329127788543701 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8056594133377075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666865348816 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7488091588020325 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6634503602981567 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9717592597007751 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8083112835884094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.598418116569519 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1635030061006546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8622384667396545 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6174925565719604 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7427422404289246 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9377859830856323 /particle/accuracy_4_of_4‚0.9443884491920471 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45145249366760254 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391682267189026 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.443735808134079 +particle/sensitivity‚0.6492762565612793 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3089715540409088 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2710605263710022 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25716453790664673 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9009732604026794 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47421813011169434 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6247660517692566 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04152993857860565 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18950073421001434 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5596442222595215 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6697631478309631 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8656843900680542 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4882562756538391 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1908876597881317 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6551626324653625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6608468294143677 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879983067512512 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20311114192008972 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9858797192573547 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7226495742797852 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2037726640701294 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_5_of_5‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7049253582954407 4 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949951171875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6743624806404114 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7759764790534973 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966437220573425 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.912625789642334 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6608468294143677 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6670882105827332 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7602262496948242 *jet/accuracy_1_of_4‚0.9687926173210144 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6312285661697388 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.627835750579834 *jet/accuracy_2_of_3‚0.5686353445053101 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9886096715927124 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984210729598999 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 _step‚844 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6547282338142395Ä}¼1×éz & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6972751021385193 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1728065460920334 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20625947415828705 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16384775936603546 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04128874093294144 +jet/accuracy_3_of_4‚0.09097576141357422 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _runtime‚4216.054624555 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.6622011661529541 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959508776664734 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18551819026470184 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22436943650245667 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 epoch‚12 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02447054721415043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8027777671813965 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9977248907089233 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998987078666687 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14644339680671692 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 +jet/accuracy_2_of_2‚0.15388260781764984 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6573659181594849 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6880514025688171 *jet/accuracy_1_of_2‚0.8004854917526245 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15792810916900635 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2881862223148346 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04650837555527687 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04718945547938347 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7724941968917847 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 $ _timestamp‚1.784501142142975e+09 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1908876597881317 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6924315690994263 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7477802038192749 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.018498167395591736 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20897305011749268 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.046843983232975006 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.021527601405978203 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7974358797073364 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7894736528396606 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940045475959778 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7436095476150513 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13790000975131989 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19338984787464142 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8455668091773987 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9148052930831909 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18362383544445038 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21298234164714813 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6845167875289917 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.047217678278684616 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46637845039367676 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6416815519332886 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5685773491859436 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012398983351886272 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5761831998825073 particle/accuracy_2_of_3‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865512847900391 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15248605608940125 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03625745698809624 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1713656187057495 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7894736528396606 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04771891608834267 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3222823143005371 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8070175051689148 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3921290934085846 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14841283857822418 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959927797317505 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 trainer/global_step‚10412 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7264717817306519 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7329127788543701 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8056594133377075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666865348816 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7488091588020325 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6634503602981567 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9717592597007751 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8083112835884094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.598418116569519 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1635030061006546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8622384667396545 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6174925565719604 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7427422404289246 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9377859830856323 /particle/accuracy_4_of_4‚0.9443884491920471 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45145249366760254 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391682267189026 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.443735808134079 +particle/sensitivity‚0.6492762565612793 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3089715540409088 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2710605263710022 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25716453790664673 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9009732604026794 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47421813011169434 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6247660517692566 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04152993857860565 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18950073421001434 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5596442222595215 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6697631478309631 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8656843900680542 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4882562756538391 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1908876597881317 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6551626324653625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6608468294143677 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879983067512512 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20311114192008972 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9858797192573547 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7226495742797852 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2037726640701294 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_5_of_5‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7049253582954407 4 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949951171875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6743624806404114 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7759764790534973 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966437220573425 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166666567325592 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.912625789642334 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6608468294143677 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6670882105827332 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.760226249694vŠDU 8242 *jet/accuracy_1_of_4‚0.9687926173210144 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6312285661697388 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.627835750579834 *jet/accuracy_2_of_3‚0.5686353445053101 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9886096715927124 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984210729598999 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 _step‚844 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6547282338142395 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5169271230697632 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8577901124954224 )loss/val_total_loss‚8.930482542674769 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18512195348739624 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9248929023742676 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3898107409477234 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7validation_average_jet_accuracy‚0.36881721019744873 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3672839403152466 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984210729598999 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6995879411697388 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5549590587615967 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2036711871623993 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 +jet/accuracy_3_of_3‚0.07163427770137787 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17184458673000336 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992790222167969 particle/accuracy_3_of_5‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18185269832611084 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46076062321662903 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04128874093294144 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5503841638565063 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7664803266525269 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8455668091773987 *jet/accuracy_2_of_4‚0.8004511594772339 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1Ì/ÝpØ‚(:å ¤§õÒªŠï+! disk.nvme0c0n1.in‚ 16.43359375! proc.memory.rssMB‚ 2858.890625 network.recv‚ 1.235329e+06% disk./.usageGB‚1170.3897514343262 disk.nvme0n1.in‚ 16.43359375& disk.nvme0n1.out‚62292.43310546875 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚24996.99951171875& disk.nvme1n1.out‚24996.99951171875 disk.dm-0.in‚ 19.68359375 disk.md0.in‚0" disk.dm-1.out‚3036.82666015625$ memory_percent‚98.82948398590088( proc.memory.availableMB‚ 191.77734375* proc.memory.percent‚17.449283599853516 cpu‚26.304694282212466 network.sent‚ 1.3926748e+07) disk./.usagePercent‚76.23447330625801 disk.nvme1n1.in‚3.25 disk.md0.out‚ 125.1796875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚62292.43310546875 proc.cpu.threads‚17# disk.dm-0.out‚83985.36962890625 disk.dm-1.in‚0 disk.rbd0.in‚0™0›Ù‚(: ¤§õÒÉšŸ, gpu.0.gpu‚27.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.156' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.718666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îhü2Ú- _wandbruntime‚4231  _runtime‚4231òÇÇÛÁ & lr-AdamW/pg2‚8.732526041633778e-05 trainer/global_step‚10449 $ _timestamp‚1.784501157706124e+09 _runtime‚4231.617334717 _step‚845 & lr-AdamW/pg1‚8.732526041633778e-05­HfÌÜÆ $ _timestamp‚1.784501157706124e+09 _runtime‚4231.617334717 _step‚845 & lr-AdamW/pg1‚8.732526041633778e-05 & lr-AdamW/pg2‚8.732526041633778e-05 trainer/global_step‚10449ÍÓæ2Ý- _wandbruntime‚4231  _runtime‚4231¦‘ì´Ú'ÞÔO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79595523 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105992693 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72927002 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403132565496832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46055727 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26871516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105992693 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410491290362880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385061406127104 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7358812888064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410491395424256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚410491395424256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405248473418856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚10449 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185598359 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7572898395648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335755759928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚403132582536192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7358812888064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335841908402 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403132565496832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46055724 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26871278 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚410491290362880 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79595521 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392634126206464 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7358724866048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0I\_ß'ßÙO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335841908402 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403132565496832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46055724 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26871278 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚410491290362880 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79595521 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392634126206464 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7358724866048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79595523 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105992693 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72927002 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403132565496832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46055727 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26871516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105992693 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410491290362880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385061406127104 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7358812888064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410491395424256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚410491395424256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405248473418856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚10449 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185598359 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7572898395648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335755759928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚403132582536192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7358812888064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680ÎQEÿê2à- _wandbruntime‚4231  _runtime‚423144ÐqáàM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392670733972480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79602842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46060017 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403169948903424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403169931864064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7573607784448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚403169948903424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72933975 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46060014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397949589726688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7336431711372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79602842 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7359402975744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403169931864064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7359402975744 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106012748 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385097101284352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26873705 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚10449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚410529334839808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397949606766048 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚847 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185615592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106002591 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7359491004928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185615592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72933719 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7336517859958 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845011579823654e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185605433 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405286021438060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26873958 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.onʪÕu _train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106002591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79602844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392670516079616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185605433 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79602844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚410529439908352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405286124626006²°â–ë&âåM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79602842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46060017 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403169948903424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403169931864064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7573607784448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚403169948903424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72933975 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46060014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397949589726688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7336431711372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79602842 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7359402975744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403169931864064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7359402975744 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106012748 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385097101284352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26873705 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚10449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚410529334839808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397949606766048 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _step‚847 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185615592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106002591 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7359491004928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185615592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72933719 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7336517859958 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845011579823654e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185605433 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405286021438060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26873958 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106002591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79602844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392670516079616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185605433 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79602844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚410529439908352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405286124626006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392670733972480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712Ï%2f 2ã- _wandbruntime‚4231  _runtime‚4231½Ø¸Jä‚ 2loss/FBt2/assignment_loss‚0.004903939959939494 /loss/FRt1/detection_loss‚0.5801780831489811 $loss/total_loss‚9.37677383365276 2loss/SRqqt2/assignment_loss‚0.5845081429473905 .loss/FBt1/detection_loss‚1.248112356024129 /loss/FBt2/detection_loss‚0.4487177287263683 _step‚848 0loss/FRt2/assignment_loss‚0.9231471206809654 2loss/SRqqt1/assignment_loss‚0.6210889130649186 /loss/FRt2/detection_loss‚1.4787629840143606 1loss/SRqqt1/detection_loss‚0.8781574808658575 epoch‚13 % _timestamp‚1.7845011579835036e+09 _runtime‚4231.894200451 0loss/FRt1/assignment_loss‚1.9770390513958125 1loss/FBt1/assignment_loss‚0.03602046666478387 1loss/SRqqt2/detection_loss‚0.5961375661592536 trainer/global_step‚10449¡£ å‡ 2loss/FBt2/assignment_loss‚0.004903939959939494 /loss/FRt1/detection_loss‚0.5801780831489811 $loss/total_loss‚9.37677383365276 2loss/SRqqt2/assignment_loss‚0.5845081429473905 .loss/FBt1/detection_loss‚1.248112356024129 /loss/FBt2/detection_loss‚0.4487177287263683 _step‚848 0loss/FRt2/assignment_loss‚0.9231471206809654 2loss/SRqqt1/assignment_loss‚0.6210889130649186 /loss/FRt2/detection_loss‚1.4787629840143606 1loss/SRqqt1/detection_loss‚0.8781574808658575 epoch‚13 % _timestamp‚1.7845011579835036e+09 _runtime‚4231.894200451 0loss/FRt1/assignment_loss‚1.9770390513958125 1loss/FBt1/assignment_loss‚0.03602046666478387 1loss/SRqqt2/detection_loss‚0.5961375661592536 trainer/global_step‚10449Ð1~Ò8kæ‚(:à ³§õÒ„‹+ network.sent‚ 1.3971008e+07 network.recv‚ 1.23949e+06% disk.nvme0n1.out‚62300.2119140625 disk.nvme1c1n1.in‚3.25 disk.dm-0.in‚ 19.68359375! disk.dm-0.out‚84000.072265625$ memory_percent‚98.83124828338623* proc.memory.percent‚17.449283599853516) disk./.usagePercent‚76.23376540379849% disk./.usageGB‚1170.3788833618164 disk.md0.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 191.48828125 cpu‚26.30327696249404 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme1n1.in‚3.25 disk.md0.out‚ 125.48828125" disk.dm-1.out‚3100.76416015625 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚62300.2119140625 disk.nvme0n1.in‚ 16.43359375( disk.nvme1c1n1.out‚25068.55224609375& disk.nvme1n1.out‚25068.55224609375! proc.memory.rssMB‚ 2858.890625.mãç‚(:„ ³§õÒéÎû+ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚173.583' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚57.861 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šë2è- _wandbruntime‚4251  _runtime‚4251"Èé trainer/global_step‚10499 % _timestamp‚1.7845011777710948e+09 _runtime‚4251.682046206 _step‚849 & lr-AdamW/pg1‚8.718289085641001e-05 & lr-AdamW/pg2‚8.718289085641001e-059‘~ÍêÇ _runtime‚4251.682046206 _step‚849 & lr-AdamW/pg1‚8.718289085641001e-05 & lr-AdamW/pg2‚8.718289085641001e-05 trainer/global_step‚10499 % _timestamp‚1.7845011777710948e+09Ñ{ýæ¡2ë- _wandbruntime‚4251  _runtime‚4251Þ‡°Ý'ì×O SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79961571 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79961573 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106499588 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386846188893184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399756321046928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7392750629888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7607866038784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7607679333888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚405000883858432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73265722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79961571 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186461161 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394453843323392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46270227 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386846163989504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚10499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412393651527680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399756338086288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚405000883858432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46270224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106489445 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73265973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7369585235336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407126009470098 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26995746 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7392750629888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412393546466304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7392662607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106489445 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚407125906282264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845011777721255e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145à‰¯.íÜO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7392662607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186451016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186461161 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394453843323392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46270227 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386846163989504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚10499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412393651527680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399756338086288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚405000883858432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46270224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106489445 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73265973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7369585235336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407126009470098 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26995746 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7392750629888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412393546466304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7392662607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106489445 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚407125906282264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845011777721255e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79961571 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79961573 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106499588 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGD&ôp"´eviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386846188893184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399756321046928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7392750629888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7607866038784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7607679333888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚405000883858432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73265722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79961571 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386881859146752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7393340717568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79968894 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106509500 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7845011780498402e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7370347335366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7608557059072 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚407163454301468 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412431696011776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79968892 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7370261186780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106499343 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79968892 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106499343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412431590943232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405038250225664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73272646 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73272407 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394490242091008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412431696011776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26998129 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚405038267265024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405038267265024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106509500 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186478394 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46274517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399793193114688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7393428746752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46274514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7393340717568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79968894 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399793210154048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚851 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394490441109504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186468235 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386881859146752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7393340717568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79968894 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106509500 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7845011780498402e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7370347335366 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7608557059072 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚407163454301468 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412431696011776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79968892 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7370261186780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106499343 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79968892 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106499343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412431590943232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405038250225664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73272646 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73272407 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394490242091008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412431696011776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26998129 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚405038267265024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405038267265024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106509500 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186478394 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46274517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399793193114688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7393428746752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46274514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Ó]2ñ- _wandbruntime‚4251  _runtime‚4251Øß"2ò trainer/global_step‚10499 _runtime‚4251.961552077 0loss/FRt1/assignment_loss‚1.9888462374607723 2loss/SRqqt2/assignment_loss‚0.6546832262074382 1loss/FBt2/assignment_loss‚0.05557837677245471 epoch‚13 0loss/FRt2/assignment_loss‚0.9325592957044903 2loss/FBt1/assignment_loss‚0.008983077566547967 /loss/FRt1/detection_loss‚0.6348698478637883 -loss/FBt2/detection_loss‚0.78622202097148 %loss/total_loss‚9.427715751878784 % _timestamp‚1.7845011780509036e+09 _step‚852 /loss/FRt2/detection_loss‚1.4468557906218311 1loss/SRqqt2/detection_loss‚0.6142938994367408 2loss/SRqqt1/assignment_loss‚0.6537576096036771 0loss/SRqqt1/detection_loss‚1.000670106344662 /loss/FBt1/detection_loss‚0.6503962633249006Wæ ó† _step‚852 /loss/FRt2/detection_loss‚1.4468557906218311 1loss/SRqqt2/detection_loss‚0.6142938994367408 2loss/SRqqt1/assignment_loss‚0.6537576096036771 0loss/SRqqt1/detection_loss‚1.000670106344662 /loss/FBt1/detection_loss‚0.6503962633249006 trainer/global_step‚10499 _runtime‚4251.961552077 0loss/FRt1/assignment_loss‚1.9888462374607723 2loss/SRqqt2/assignment_loss‚0.6546832262074382 1loss/FBt2/assignment_loss‚0.05557837677245471 epoch‚13 0loss/FRt2/assignment_loss‚0.9325592957044903 2loss/FBt1/assignment_loss‚0.008983077566547967 /loss/FRt1/detection_loss‚0.6348698478637883 -loss/FBt2/detection_loss‚0.78622202097148 %loss/total_loss‚9.427715751878784 % _timestamp‚1.7845011780509036e+09Ô7 iô‚(:Þ §õÒ‘­ó+ disk.nvme1c1n1.in‚3.25" disk.dm-1.out‚3101.00537109375! proc.memory.rssMB‚ 2858.890625* proc.memory.percent‚17.449283599853516 network.sent‚ 1.4046982e+07% disk./.usageGB‚1167.4780883789062! disk.nvme0c0n1.in‚ 16.43359375 disk.nvme1n1.in‚3.25$ disk.nvme1n1.out‚25069.517578125$ memory_percent‚98.83148670196533( proc.memory.availableMB‚ 191.44921875 network.recv‚ 1.242773e+06( disk.nvme0c0n1.out‚62302.91259765625 disk.nvme0n1.in‚ 16.43359375! disk.dm-0.out‚84002.244140625 cpu‚26.30179410775832 proc.cpu.threads‚17& disk.nvme1c1n1.out‚25069.517578125 disk.md0.in‚0 disk.md0.out‚ 126.0703125 disk.dm-0.in‚ 19.68359375 disk.dm-1.in‚0) disk./.usagePercent‚76.04481930492794 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚62302.91259765625dpÍíõ‚(:„ §õÒ¾öß+ gpu.0.gpu‚31.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚116.574' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.858 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ªG§2ö- _wandbruntime‚4272  _runtime‚4272©³GÈ÷ & lr-AdamW/pg1‚8.703993981140183e-05 & lr-AdamW/pg2‚8.703993981140183e-05 trainer/global_step‚10549 % _timestamp‚1.7845011981716104e+09 _runtime‚4272.082564267 _step‚853jÿÔ©ÍøÇ trainer/global_step‚10549 % _timestamp‚1.7845011981716104e+09 _runtime‚4272.082564267 _step‚853 & lr-AdamW/pg1‚8.703993981140183e-05 & lr-AdamW/pg2‚8.703993981140183e-05ÕîÊ2ù- _wandbruntime‚4272  _runtime‚4272 Ñn:¯úÖO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414295802569728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚10549 _runtime‚4272.083540458 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7642841917952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7426600349696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73603957 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414295907631104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406869219259392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73603719 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388630946755584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80327623 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409003442333506 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7642659407360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406869202220032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396273788673536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187323963 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46484724 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7403414710744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚401599941474288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414295802569728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚10549 _runtime‚4272.083540458 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7642841917952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7426600349696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73603957 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414295907631104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406869219259392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73603719 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388630946755584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80327623 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409003442333506 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7642659407360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406869202220032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396273788673536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187323963 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46484724 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7403414710744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚401599941474288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187341196 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80334944 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7643363183616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7404176810774 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401636796502688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚855 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388666617009152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414333952115200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406906568587264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7427366488576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107006252 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406906585626624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80334942 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7427278459392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73610470 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚4272.360016357 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414333952115200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414333847046656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46489014 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396310181308416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187331037 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106996095 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106996095-!±q¶þæM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80334944 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7643363183616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7404176810774 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401636796502688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚855 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388666617009152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414333952115200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406906568587264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7427366488576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107006252 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406906585626624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80334942 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7427278459392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73610470 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚4272.360016357 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414333952115200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414333847046656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46489014 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396310181308416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187331037 K=DeviceStatsMonú#Ð96itor.on_train_batch_end/active.small_pool.freed‚ 106996095 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106996095 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46489017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27121456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406906568587264 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187341196 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464×èø‰2ÿ- _wandbruntime‚4272  _runtime‚4272x×–€ /loss/FBt2/detection_loss‚0.4158637481741607 %loss/total_loss‚9.540941625193957 _step‚856 .loss/FRt2/assignment_loss‚0.92578437264639 2loss/SRqqt1/assignment_loss‚0.9107840844892627 /loss/FRt1/detection_loss‚0.5842073663466858 /loss/FRt2/detection_loss‚1.3936221665784518 2loss/FBt2/assignment_loss‚0.020163013599505532 epoch‚13 trainer/global_step‚10549 2loss/SRqqt2/assignment_loss‚0.5606681688902648 1loss/FBt1/assignment_loss‚0.13783071523264567 $ _timestamp‚1.784501198450026e+09 _runtime‚4272.364396664 0loss/FRt1/assignment_loss‚2.0557767766326727 1loss/SRqqt1/detection_loss‚1.0927268450859149 1loss/SRqqt2/detection_loss‚0.5465235830158801 /loss/FBt1/detection_loss‚0.89699078450212268¶9 † 0loss/FRt1/assignment_loss‚2.0557767766326727 1loss/SRqqt1/detection_loss‚1.0927268450859149 1loss/SRqqt2/detection_loss‚0.5465235830158801 /loss/FBt1/detection_loss‚0.8969907845021226 /loss/FBt2/detection_loss‚0.4158637481741607 %loss/total_loss‚9.540941625193957 _step‚856 .loss/FRt2/assignment_loss‚0.92578437264639 2loss/SRqqt1/assignment_loss‚0.9107840844892627 /loss/FRt1/detection_loss‚0.5842073663466858 /loss/FRt2/detection_loss‚1.3936221665784518 2loss/FBt2/assignment_loss‚0.020163013599505532 epoch‚13 trainer/global_step‚10549 2loss/SRqqt2/assignment_loss‚0.5606681688902648 1loss/FBt1/assignment_loss‚0.13783071523264567 $ _timestamp‚1.784501198450026e+09 _runtime‚4272.364396664Ø–/‰š‚‚(: ѧõÒÔÄ+ gpu.0.gpu‚52.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.459' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.81966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤kã©hƒ‚(:Ý ѧõÒ‚™Ø,% disk./.usageGB‚1170.3887252807617 disk.nvme1c1n1.in‚3.25( disk.nvme1c1n1.out‚25071.63720703125! disk.dm-0.out‚87551.412109375" disk.dm-1.out‚3101.09912109375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚65851.0205078125$ memory_percent‚98.83308410644531 network.sent‚ 1.4094453e+07 network.recv‚ 1.246711e+06 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 16.43359375$ proc.memory.availableMB‚191.1875 proc.cpu.threads‚17) disk./.usagePercent‚76.23440646674885 disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.md0.out‚ 126.51953125 disk.dm-1.in‚0* proc.memory.percent‚17.449283599853516 disk.nvme0n1.in‚ 16.43359375% disk.nvme0n1.out‚65851.0205078125& disk.nvme1n1.out‚25071.63720703125 disk.dm-0.in‚ 19.68359375! proc.memory.rssMB‚ 2858.890625 cpu‚26.30032021809548Û¥4›„‚(: à§õÒ÷£§+ gpu.0.gpu‚78.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚192.718' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.23933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒôC÷g…‚(:Ü à§õÒºÆý+! disk.nvme0c0n1.in‚ 16.43359375' disk.nvme0c0n1.out‚65851.9033203125 disk.md0.in‚0 disk.dm-0.in‚ 19.68359375! disk.dm-0.out‚87552.169921875 disk.nvme0n1.in‚ 16.43359375 disk.md0.out‚ 126.7265625 disk.dm-1.in‚0 network.sent‚ 1.4159036e+07 network.recv‚ 1.250593e+06% disk./.usageGB‚1170.3898963928223& disk.nvme1c1n1.out‚25072.146484375$ disk.nvme1n1.out‚25072.146484375 disk.rbd0.in‚0! disk.dm-1.out‚3101.2724609375% disk.nvme0n1.out‚65851.9033203125$ memory_percent‚98.83038997650146( proc.memory.availableMB‚ 191.62890625! proc.memory.rssMB‚ 2858.890625* proc.memory.percent‚17.449283599853516 cpu‚26.298991902547446 proc.cpu.threads‚17) disk./.usagePercent‚76.23448274827047 disk.nvme1c1n1.in‚3.25 disk.nvme1n1.in‚3.25 disk.rbd0.out‚0~gª2†- _wandbruntime‚4292  _runtime‚4292º¤îoƇÀ % lr-AdamW/pg1‚8.68964095709644e-05 % lr-AdamW/pg2‚8.68964095709644e-05 trainer/global_step‚10599 % _timestamp‚1.7845012185931973e+09 _runtime‚4292.504156413 _step‚857nµ¯aˈŠtrainer/global_step‚10599 % _timestamp‚1.7845012185931973e+09 _runtime‚4292.504156413 _step‚857 % lr-AdamW/pg1‚8.68964095709644e-05 % lr-AdamW/pg2‚8.68964095709644e-05ÙýÒê2‰- _wandbruntime‚4292  _runtime‚4292öRåÝ'Š×O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416198058673152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188186765 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚410880875196914 % _timestamp‚1.7845012185942729e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416198058673152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7460538091520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107493092 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416198163734528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73940601 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107482949 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398093464954880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403443544862288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390415704617984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7437330334626 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7677575729152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80693671 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107482949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928QÇgþâ'‹ÜO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7677575729152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80693671 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107482949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416198058673152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188186765 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚410880875196914 % _timestamp‚1.7845012185942729e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416198058673152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7460538091520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107493092 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416198163734528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73940601 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107482949 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398093464954880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403443544862288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390415704617984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7437330334626 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152ÚaKAÎ2Œ- _wandbruntime‚4292  _runtime‚4292HI™+áM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7438006286182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390451399775232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188203998 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410918423216230 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚10599 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398129861028352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7461216201216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚859 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408774886948864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107492847 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410918320028284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚416236208218624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845012188707638e+09 _runtime‚4292.781839929 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107503004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107492847 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80700994 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7437920137596 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188203998 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408774903988224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46703517 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚398129649426944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =›ÉG¼XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188193839 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46703514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27243833 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7678461253120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73947347 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408774886948864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107503004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚416236208218624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403480399890688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80700994 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73947617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27244100 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408774903988224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚416236103150080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188193839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7461304230400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7678274555392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482Òd5Öì&ŽæM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408774886948864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107503004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚416236208218624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403480399890688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80700994 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73947617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27244100 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408774903988224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚416236103150080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188193839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7461304230400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7678274555392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7438006286182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390451399775232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188203998 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410918423216230 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚10599 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398129861028352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7461216201216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚859 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408774886948864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107492847 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410918320028284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚416236208218624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845012188707638e+09 _runtime‚4292.781839929 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107503004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107492847 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80700994 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7437920137596 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188203998 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408774903988224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46703517 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚398129649426944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188193839 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46703514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27243833 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7678461253120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73947347 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722Û<Ø„2- _wandbruntime‚4292  _runtime‚4292íU_ „ 0loss/FRt1/assignment_loss‚2.1652356045034997 3loss/FBt2/assignment_loss‚0.0005008423227727121 1loss/SRqqt1/detection_loss‚1.1219241808134097 %loss/total_loss‚10.09878500417981 _step‚860 0loss/FRt2/assignment_loss‚0.9104827606243643 2loss/SRqqt1/assignment_loss‚0.7431573465190552 2loss/SRqqt2/assignment_loss‚0.5169519031793787 /loss/FRt1/detection_loss‚0.5827732267756071 /loss/FRt2/detection_loss‚1.4101752051386063 1loss/SRqqt2/detection_loss‚0.5983763247280829 epoch‚13 trainer/global_step‚10599 /loss/FBt1/detection_loss‚1.2677895997961361 /loss/FBt2/detection_loss‚0.7700262358784675 _runtime‚ 4292.78255381 1loss/FBt1/assignment_loss‚0.01139177390042884 % _timestamp‚1.7845012188718333e+09g†¡‘‰ 2loss/SRqqt1/assignment_loss‚0.7431573465190552 2loss/SRqqt2/assignment_loss‚0.5169519031793787 /loss/FRt1/detection_loss‚0.5827732267756071 /loss/FRt2/detection_loss‚1.4101752051386063 1loss/SRqqt2/detection_loss‚0.5983763247280829 epoch‚13 trainer/global_step‚10599 /loss/FBt1/detection_loss‚1.2677895997961361 /loss/FBt2/detection_loss‚0.7700262358784675 _runtime‚ 4292.78255381 1loss/FBt1/assignment_loss‚0.01139177390042884 % _timestamp‚1.7845012188718333e+09 0loss/FRt1/assignment_loss‚2.1652356045034997 3loss/FBt2/assignment_loss‚0.0005008423227727121 1loss/SRqqt1/detection_loss‚1.1219241808134097 %loss/total_loss‚10.09878500417981 _step‚860 0loss/FRt2/assignment_loss‚0.9104827606243643Ü%…~¿h’‚(:Ý ï§õÒÞ¢²+ disk.nvme0n1.in‚ 16.43359375 disk.nvme1c1n1.in‚3.25& disk.nvme1n1.out‚25073.87841796875# disk.dm-0.out‚87565.92236328125$ memory_percent‚98.83208274841309' proc.memory.availableMB‚ 191.3515625 network.recv‚ 1.254348e+06) disk./.usagePercent‚76.23392467564004% disk./.usageGB‚1170.3813285827637' disk.nvme0c0n1.out‚65864.8017578125! proc.memory.rssMB‚ 2858.890625 network.sent‚ 1.4162e+07 disk.rbd0.in‚0 disk.nvme1n1.in‚3.25 disk.md0.in‚0* proc.memory.percent‚17.449283599853516 cpu‚26.29743973351336 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚65864.8017578125( disk.nvme1c1n1.out‚25073.87841796875 disk.md0.out‚ 127.0703125 disk.dm-0.in‚ 19.68359375 proc.cpu.threads‚17! disk.dm-1.out‚3101.3818359375! disk.nvme0c0n1.in‚ 16.43359375ߊà)“‚(:„ ï§õÒÎíŽ, gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚199.518' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹wØ 2”- _wandbruntime‚4312  _runtime‚4312Pz…™È•Â trainer/global_step‚10649 % _timestamp‚1.7845012389760907e+09 _runtime‚4312.887032967 _step‚861 & lr-AdamW/pg1‚8.675230243402585e-05 & lr-AdamW/pg2‚8.675230243402585e-05 V¥Í–Ç & lr-AdamW/pg2‚8.675230243402585e-05 trainer/global_step‚10649 % _timestamp‚1.7845012389760907e+09 _runtime‚4312.887032967 _step‚861 & lr-AdamW/pg1‚8.675230243402585e-05ÝÿAF2—- _wandbruntime‚4312  _runtime‚4312Pˆ LÝ'˜×O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7712667950592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7494563855360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7712479148544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81059723 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7494475833344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7494475833344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392200462480384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚405287148250288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418100314776576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7471073661560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚410605838943232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46913724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27364489 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189049567 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74277961 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399913130430976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412758308060322 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7471159810034 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚410605838943232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚418100314776576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27364237 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399912916725248 trainer/global_step‚10649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392200437576704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374Âꡃå™ÜO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7712479148544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81059723 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7494475833344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7494475833344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392200462480384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚405287148250288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418100314776576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7471073661560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482˜–ý JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚410605838943232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46913724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27364489 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189049567 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74277961 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399913130430976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412758308060322 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7471159810034 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚410605838943232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚418100314776576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27364237 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399912916725248 trainer/global_step‚10649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392200437576704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7712667950592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7494563855360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248Þ­ÿm*2š- _wandbruntime‚4313  _runtime‚4313G«™cç&›áM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7713177171968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189056641 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46918017 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399949309905920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471835761590 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27366686 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410643205310464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46918014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7495153943040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410643205310464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7713359675392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189066800 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107989599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚10649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚418138464322048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27366938 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7495241972224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7495241972224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74284700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410643222349824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410643222349824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107989599 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7495153943040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405324020318048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405324003278688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7471749613004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412795752891692 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81067042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚863 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81067044 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0r;ºì&œæM B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81067044 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7713177171968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189056641 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46918017 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399949309905920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471835761590 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27366686 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410643205310464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46918014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7495153943040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410643205310464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7713359675392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189066800 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107989599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚10649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚418138464322048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27366938 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7495241972224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7495241972224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74284700 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410643222349824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410643222349824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107989599 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7495153943040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405324020318048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405324003278688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7471749613004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412795752891692 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81067042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚863ßV>02- _wandbruntime‚4313  _runtime‚4313òUCe žƒ 2loss/SRqqt2/assignment_loss‚0.5519493712595261 2loss/FBt1/assignment_loss‚0.020720059606040713 2loss/FBt2/assignment_loss‚0.001529520833561328 /loss/FRt2/detection_loss‚1.4378783046285768 1loss/SRqqt1/detection_loss‚0.9821307151155039 /loss/FBt2/detection_loss‚0.5864886425050169 /loss/FRt1/detection_loss‚0.6382788698560208 %loss/total_loss‚8.928074259974526 trainer/global_step‚10649 % _timestamp‚1.7845012392513611e+09 _runtime‚ 4313.16205274 0loss/FRt2/assignment_loss‚0.8359802141500159 1loss/SRqqt2/detection_loss‚0.6492560922158314 /loss/FBt1/detection_loss‚0.3792112067652245 _step‚864 /loss/FRt1/assignment_loss‚2.137173990708963 2loss/SRqqt1/assignment_loss‚0.7074772723302427 epoch‚13屪bŸˆ %loss/total_loss‚8.928074259974526 trainer/global_step‚10649 % _timestamp‚1.7845012392513611e+09 _runtime‚ 4313.16205274 0loss/FRt2/assignment_loss‚0.8359802141500159 1loss/SRqqt2/detection_loss‚0.6492560922158314 /loss/FBt1/detection_loss‚0.3792112067652245 _step‚864 /loss/FRt1/assignment_loss‚2.137173990708963 2loss/SRqqt1/assignment_loss‚0.7074772723302427 epoch‚13 2loss/SRqqt2/assignment_loss‚0.5519493712595261 2loss/FBt1/assignment_loss‚0.020720059606040713 2loss/FBt2/assignment_loss‚0.001529520833561328 /loss/FRt2/detection_loss‚1.4378783046285768 1loss/SRqqt1/detection_loss‚0.9821307151155039 /loss/FBt2/detection_loss‚0.5864886425050169 /loss/FRt1/detection_loss‚0.6382788698560208à¶^Š’m ‚(:â þ§õÒ¾Èð+ network.recv‚ 1.258639e+06% disk./.usageGB‚1166.9132957458496 disk.nvme0n1.in‚ 16.43359375& disk.nvme1n1.out‚25075.41552734375! proc.memory.rssMB‚ 2858.890625* proc.memory.percent‚17.449283599853516& disk.nvme0n1.out‚65868.62451171875 network.sent‚ 1.4237624e+07) disk./.usagePercent‚76.00803098816803# disk.dm-0.out‚87570.31787109375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 16.43359375( disk.nvme0c0n1.out‚65868.62451171875 disk.nvme1c1n1.in‚3.25 disk.dm-1.in‚0" disk.dm-1.out‚3101.51513671875 disk.nvme1n1.in‚3.25 disk.md0.in‚0 disk.md0.out‚ 127.45703125$ memory_percent‚98.83079528808594 proc.cpu.threads‚17 disk.dm-0.in‚ 19.68359375 disk.rbd0.in‚0( disk.nvme1c1n1.out‚25075.41552734375$ proc.memory.availableMB‚191.5625 cpu‚26.296057719155737 bÌš¡‚(: þ§õÒ¥úœ, gpu.0.gpu‚67.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.294' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.76466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z˜©=2¢- _wandbruntime‚4332  _runtime‚4332ËøYSȣ & lr-AdamW/pg1‚8.660762070875445e-05 & lr-AdamW/pg2‚8.660762070875445e-05 trainer/global_step‚10699 % _timestamp‚1.7845012590062335e+09 _runtime‚4332.917195058 _step‚865ã‘¶Í¤Ç & lr-AdamW/pg2‚8.660762070875445e-05 trainer/global_step‚10699 % _timestamp‚1.7845012590062335e+09 _runtime‚4332.917195058 _step‚865 & lr-AdamW/pg1‚8.660762070875445e-05á¡YÃ'2¥- _wandbruntime‚4332  _runtime‚4332'«¶áS¦ÖO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7528501597184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7528413575168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81425771 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412474157304832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7528501597184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108486596 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407130734598928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108476453 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420002675941376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420002675941376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412474157304832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420002570880000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108486596 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393985195439104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚414635740923730 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7504903136968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189912369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74615243 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81425771 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393985220342784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845012590072985e+09ð¬ÈÃá'§ÛO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81425771 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393985220342784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845012590072985e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7528501597184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7528413575168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81425771 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412474157304832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7528501597184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108486596 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407130734598928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108476453 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420002675941376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420002675941376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412474157304832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420002570880000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108486596 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393985195439104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚414635740923730 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7504903136968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189912369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74615243 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152âó¦ïK2¨- _wandbruntime‚4333  _runtime‚4333òrù›å&©ßM _runtime‚4333.194310652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74622010 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7748100662272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189919443 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420040720425472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401768991258624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7505665236998 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108486351 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412511523672064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81433092 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420040720425472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189919443 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108496508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47132514 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394020890596352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414673288943046 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7529179714048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 _step‚867 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27489493 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108496508 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7529091684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407167606666688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407167623706048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81433092 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚414673185755100 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 # _timestamp‚1.78450125928323e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81433094 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7529091684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47132517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10699 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412511540711424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81433094 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74621756 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189929602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238‹M¬•\ªäM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189929602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _runtime‚4333.194310652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74622010 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7748100662272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189919443 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420040720425472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401768991258624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7505665236998 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108486351 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412511523672064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81433092 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420040720425472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189919443 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108496508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47132514 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394020890596352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414673288943046 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7529179714048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.smN“z+Žall_pool.peak‚487 _step‚867 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27489493 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108496508 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7529091684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407167606666688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407167623706048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81433092 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚414673185755100 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 # _timestamp‚1.78450125928323e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81433094 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7529091684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47132517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10699 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412511540711424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81433094 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74621756 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414342492705792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403552301770240 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414342475666432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81791823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81791821 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421904932044800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845012793805096e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408974355026288 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108983348 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421904826983424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395769953301504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚408974337986928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4353.291450642 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414342492705792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81791823 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7562439339008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190765026 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7562439339008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7538732612376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚870 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108983348 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421904826983424 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414342492705792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403552301770240 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414342475666432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81791823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81791821 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421904932044800 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845012793805096e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408974355026288 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108983348 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421904826983424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395769953301504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚408974337986928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4353.291450642 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414342492705792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81791823 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7562439339008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190765026 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7562439339008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7538732612376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚870 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108983348 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421904826983424 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74959361 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7539408563820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421942976528896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81799142 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403588910730240 # _timestamp‚1.78450127965785e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74959619 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7539494712406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414379842033664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190782245 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚4353.568913338 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108993260 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190792404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27612347 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7563029426688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421942976528896 _KDeviceStatsMoniÀ3xtor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395805648458752 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7563117455872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7783046475776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108983103 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7563117455872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409011227094048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190782245 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416550721806454 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81799144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7563029426688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27612602 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409011210054688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47347017 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403588694934528 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108983103 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152Í“3Ëê&ºäM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81799144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421942871460352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416550618618508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81799142 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108983103 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7563117455872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409011227094048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190782245 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416550721806454 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81799144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7563029426688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27612602 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409011210054688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47347017 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403588694934528 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108983103 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190792404 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108993260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74959361 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7539408563820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421942976528896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81799142 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403588910730240 # _timestamp‚1.78450127965785e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74959619 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7539494712406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414379842033664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190782245 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚4353.568913338 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108993260 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190792404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27612347 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7563029426688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421942976528896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395805648458752 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7563117455872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7783046475776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423807083086848 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416210811067392 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109480100 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82157873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405371994686464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7572562087784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚874 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10799 % _timestamp‚1.7845012997412686e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82157871 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109469957 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47557227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418390606650546 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27733483 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27733262 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109469957 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7596289058816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47557224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410817941374928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397554736067584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75290486 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423807188148224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7596377080832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75290710 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418390503462712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423807188148224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416210811067392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191637973 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7596377080832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82157871½qEm ÅÜO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397554711163904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚410817958414288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7572648236258 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27733262 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109469957 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7596289058816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47557224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410817941374928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397554736067584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75290486 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423807188148224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 759637î®ðu7080832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75290710 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418390503462712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423807188148224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416210811067392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191637973 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7596377080832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82157871 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109480100 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423807083086848 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416210811067392 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109480100 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82157873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405371994686464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7572562087784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚874 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10799 % _timestamp‚1.7845012997412686e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82157871 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109469957 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47557227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418390606650546 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27733483 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75297151 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410854813442688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚416248177434624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397590406321152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191655206 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418428154669862 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191655206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47561517 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109490012 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423845232632320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7597055197696 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410854830482048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7817980788224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416248177434624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7573324187814 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423845127563776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚10799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191645047 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416248160395264 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405408586127872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7597055197696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109479855 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109479855 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423845232632320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7573238039228 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416248160395264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚875 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82165194 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845013000165317e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27735637 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7596967168512E»ì&ÈæM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75297151 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410854813442688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚416248177434624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397590406321152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191655206 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418428154669862 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191655206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47561517 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109490012 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423845232632320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7597055197696 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410854830482048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7817980788224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416248177434624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7573324187814 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423845127563776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚10799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191645047 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416248160395264 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405408586127872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7597055197696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109479855 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109479855 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423845232632320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7573238039228 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416248160395264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚875 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82165194 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845013000165317e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27735637 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7596967168512ëÓ@W2É- _wandbruntime‚4373  _runtime‚4373Å— IÊ /loss/FBt2/detection_loss‚0.8761677239866307 0loss/FRt2/assignment_loss‚1.0179835957816885 /loss/FRt1/detection_loss‚0.5969919928649983 1loss/SRqqt1/detection_loss‚1.0715514179579653 1loss/SRqqt1/assignment_loss‚0.745176232620582 0loss/FBt1/assignment_loss‚0.0908249025478897 %loss/total_loss‚9.763709749541214 trainer/global_step‚10799 _step‚876 epoch‚13 0loss/FRt1/assignment_loss‚2.1146028207545458 2loss/SRqqt2/assignment_loss‚0.6449204426098181 1loss/FBt2/assignment_loss‚0.03047400286978601 /loss/FRt2/detection_loss‚1.5213283006188298 1loss/SRqqt2/detection_loss‚0.5883070364094108 $ _timestamp‚1.784501300017578e+09 _runtime‚4373.928257344 0loss/FBt1/detection_loss‚0.46538128051906824¸: 9 ˆ 0loss/FRt1/assignment_loss‚2.1146028207545458 2loss/SRqqt2/assignment_loss‚0.6449204426098181 1loss/FBt2/assignment_loss‚0.03047400286978601 /loss/FRt2/detection_loss‚1.5213283006188298 1loss/SRqqt2/detection_loss‚0.5883070364094108 $ _timestamp‚1.784501300017578e+09 _runtime‚4373.928257344 0loss/FBt1/detection_loss‚0.46538128051906824 /loss/FBt2/detection_loss‚0.8761677239866307 0loss/FRt2/assignment_loss‚1.0179835957816885 /loss/FRt1/detection_loss‚0.5969919928649983 1loss/SRqqt1/detection_loss‚1.0715514179579653 1loss/SRqqt1/assignment_loss‚0.745176232620582 0loss/FBt1/assignment_loss‚0.0908249025478897 %loss/total_loss‚9.763709749541214 trainer/global_step‚10799 _step‚876 epoch‚13ì`ŠoeoÌ‚(:ä º¨õÒŽÚ+$ disk.nvme1n1.out‚25152.525390625! disk.nvme0c0n1.in‚ 26.22265625& disk.nvme0c0n1.out‚68333.833984375# proc.memory.rssMB‚ 2862.66015625$ disk./.usageGB‚1167.603199005127 disk.nvme1c1n1.in‚ 5.2109375# disk.dm-0.out‚90089.90380859375* proc.memory.percent‚17.472290992736816 cpu‚26.290479931919585 proc.cpu.threads‚17 disk.nvme0n1.in‚ 26.22265625 disk.dm-0.in‚ 31.43359375 disk.dm-1.in‚0 disk.rbd0.in‚0 network.sent‚ 1.4415608e+07& disk.nvme1c1n1.out‚25152.525390625 disk.md0.in‚0 disk.md0.out‚ 134.953125" disk.dm-1.out‚3109.02490234375 disk.rbd0.out‚0# memory_percent‚98.7342357635498' proc.memory.availableMB‚ 207.3828125 network.recv‚ 1.274166e+06( disk./.usagePercent‚76.0529685070918$ disk.nvme0n1.out‚68333.833984375 disk.nvme1n1.in‚ 5.2109375âêÝçšÍ‚(: º¨õÒ›þ‚, gpu.0.gpu‚51.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.962' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.98733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mf2Î- _wandbruntime‚4393  _runtime‚4393álOÈÏ & lr-AdamW/pg1‚8.617015122245145e-05 & lr-AdamW/pg2‚8.617015122245145e-05 trainer/global_step‚10849 % _timestamp‚1.7845013200867014e+09 _runtime‚4393.998385625 _step‚877žâ”’ÍÐÇ & lr-AdamW/pg2‚8.617015122245145e-05 trainer/global_step‚10849 % _timestamp‚1.7845013200867014e+09 _runtime‚4393.998385625 _step‚877 & lr-AdamW/pg1‚8.617015122245145e-05íŽT#2Ñ- _wandbruntime‚4393  _runtime‚4393¥°]sÝÒ×O YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418079112389632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27856935 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7852430175744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47771727 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7606477711666 % _timestamp‚1.7845013200874312e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4393.998444115 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425709444251648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82523923 JBDevÄ¢‚D!iceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚412661544762928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚878 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7606391563192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109966709 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109976852 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7630314822656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420267936326120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82523923 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425709339190272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418079129428992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75628917 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407191714594304 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425709444251648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448—Ú;ƒâ'ÓÜO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425709339190272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418079129428992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75628917 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407191714594304 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425709444251648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418079112389632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27856935 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7852430175744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47771727 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7606477711666 % _timestamp‚1.7845013200874312e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4393.998444115 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425709444251648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82523923 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚412661544762928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚878 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7606391563192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109966709 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109976852 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7630314822656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420267936326120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82523923 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482î"·ßƒ2Ô- _wandbruntime‚4394  _runtime‚4394ÇH³èæ&ÕàM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚879 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399375164183552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75635417 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚ 4394.27680797 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7630904910336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407228106607616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7852942424064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420305484345324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418116495796224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82531242 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192518008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82531244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27859403 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚412698433870048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7630992939520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27859659 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192507849 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82531242 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7607153663222 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412698416830688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418116495796224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82531244 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399375189087232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7853133316096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7630904910336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚418116478756864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109976607 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192518008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47776014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7630992939520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109986764 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109986764 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47776017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407228322403328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚418116478756864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946AkãÖåM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418116495796224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82531244 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399375189087232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7853133316096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7630904910336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚418116478756864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109976607 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192518008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47776014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7630992939520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109986764 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109986764 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47776017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407228322403328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4181164787568ðñ²64 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚879 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399375164183552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75635417 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚ 4394.27680797 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7630904910336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407228106607616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7852942424064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420305484345324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418116495796224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82531242 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192518008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82531244 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27859403 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚412698433870048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7630992939520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27859659 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192507849 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82531242 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7607153663222 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412698416830688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47986227 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427611700355072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409011667829760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7664164542464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427611595293696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚422145472377362 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7640221038600 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47986224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110473604 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚4414.296892435 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845013403859584e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110473604 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193363577 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75966505 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419947447790592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7664164542464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27980281 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193353432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7887239818240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414505165190288 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419947430751232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427611700355072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27980532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82889973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7887416037376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419947430751232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427611595293696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚422145369189528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409011466706944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110463461 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7664252564480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427611595293696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚422145369189528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409011466706944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110463461 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7664252564480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47986227 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427611700355072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409011667829760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7664164542464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427611595293696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚422145472377362 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7640221038600 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47986224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110473604 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚4414.296892435 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845013403859584e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110473604 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193363577 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75966505 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419947447790592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7664164542464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27980281 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193353432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7887239818240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414505165190288 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419947430751232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427611700355072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27980532ò_~Ó|2ä- _wandbruntime‚4414  _runtime‚4414AT”GåáM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110473359 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193380810 \HDeviceStatsMonitor.on_train_batch_end/inactive_spliµ\  t_bytes.all.allocated‚409048067614208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7664842652160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409047856012800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7640983138630 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7664930681344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82897294 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82897292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75973433 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427649639770624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427649744839168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845013406643496e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4414.575459074 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401159922045952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7888120664576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110473359 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82897292 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82897294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7664842652160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚883 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193370651 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7640896990044 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47990517 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419984814157824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7664930681344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414542037258048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419984797118464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193370651 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75973166 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47990514 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427649744839168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414542020218688 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193380810 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409048067614208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7664842652160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409047856012800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7640983138630 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7664930681344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82897294 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82897292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75973433 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427649639770624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427649744839168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845013406643496e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4414.575459074 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401159922045952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7888120664576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110473359 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82897292 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82897294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7664842652160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚883 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193370651 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7640896990044 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47990517 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419984814157824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7664930681344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414542037258048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419984797118464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193370651 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75973166 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47990514 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427649744839168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414542020218688 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110473359ó+ J62ç- _wandbruntime‚4414  _runtime‚44147ÇÄ*è _runtime‚4414.576248815 2loss/SRqqt2/assignment_loss‚0.5871364290575216 3loss/FBt1/assignment_loss‚0.0032647022388352957 epoch‚13 trainer/global_step‚10899 _step‚884 .loss/FRt1/detection_loss‚0.589446235462084 /loss/FBt1/detection_loss‚0.6950848645976057 #loss/total_loss‚8.8588390914388 % _timestamp‚1.7845013406654267e+09 2loss/SRqqt1/assignment_loss‚0.6567554824700158 .loss/FRt2/detection_loss‚1.461174705125154 1loss/SRqqt1/detection_loss‚0.9959883314196861 /loss/FRt1/assignment_loss‚2.115766756871634 0loss/FRt2/assignment_loss‚0.7746829043821509 1loss/FBt2/assignment_loss‚0.00121084658565059 1loss/SRqqt2/detection_loss‚0.6716548066076083 0loss/FBt2/detection_loss‚0.30667302662085244" £Ð é† /loss/FRt1/assignment_loss‚2.115766756871634 0loss/FRt2/assignment_loss‚0.7746829043821509 1loss/FBt2/assignment_loss‚0.00121084658565059 1loss/SRqqt2/detection_loss‚0.6716548066076083 0loss/FBt2/detection_loss‚0.30667302662085244 _runtime‚4414.576248815 2loss/SRqqt2/assignment_loss‚0.5871364290575216 3loss/FBt1/assignment_loss‚0.0032647022388352957 epoch‚13 trainer/global_step‚10899 _step‚884 .loss/FRt1/detection_loss‚0.589446235462084 /loss/FBt1/detection_loss‚0.6950848645976057 #loss/total_loss‚8.8588390914388 % _timestamp‚1.7845013406654267e+09 2loss/SRqqt1/assignment_loss‚0.6567554824700158 .loss/FRt2/detection_loss‚1.461174705125154 1loss/SRqqt1/detection_loss‚0.9959883314196861ô?·pê‚(:å ç¨õÒ¡ñõ+ disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 26.22265625 disk.nvme1c1n1.in‚ 5.2109375$ memory_percent‚99.12939071655273# proc.memory.rssMB‚ 2862.66015625$ disk.nvme1n1.out‚25155.681640625 disk.md0.in‚0 disk.dm-0.in‚ 31.43359375# disk.dm-0.out‚90095.77001953125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 26.22265625& disk.nvme0n1.out‚68340.56396484375& disk.nvme1c1n1.out‚25155.681640625$ disk./.usageGB‚1167.603904724121! disk.dm-1.out‚3110.1455078125( disk.nvme0c0n1.out‚68340.56396484375& proc.memory.availableMB‚ 142.640625 cpu‚26.286348977294242 network.sent‚ 1.455876e+07 disk.nvme1n1.in‚ 5.2109375 disk.rbd0.out‚0* proc.memory.percent‚17.472290992736816 proc.cpu.threads‚17 network.recv‚ 1.286263e+06) disk./.usagePercent‚76.05301447478399 disk.md0.out‚136.3125çâ›ë‚(: ç¨õÒ…­ž, gpu.0.gpu‚82.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚167.972' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚55.99066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾pÆ 2ì- _wandbruntime‚4434  _runtime‚4434ÚÓô<Èí & lr-AdamW/pg1‚8.587567468543682e-05 & lr-AdamW/pg2‚8.587567468543682e-05 trainer/global_step‚10949 % _timestamp‚1.7845013607361636e+09 _runtime‚4434.647114889 _step‚885¤‰!ÍîÇ _step‚885 & lr-AdamW/pg1‚8.587567468543682e-05 & lr-AdamW/pg2‚8.587567468543682e-05 trainer/global_step‚10949 % _timestamp‚1.7845013607361636e+09 _runtime‚4434.647114889õÊ_A2ï- _wandbruntime‚4434  _runtime‚4434¨Y9hÝ'ð×O XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48200727 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76304003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28103516 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424022802052936 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110960213 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429513851397120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402908984751104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110970356 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421815766152192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194226379 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83256023 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110960213 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416348768578288 _step‚886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429513956458496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429513956458496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429513851397120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28103279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410831160310272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7922358069760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424022905240770 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421815749112832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7698190306304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7698102284288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7845013607371733e+09 _runtime‚4434.649129303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76304243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7922175559168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83256023 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚421815766152192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48200724 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421815749112832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7698190306304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110970356 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722Å^3áóñÜO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110970356 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421815766152192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194226379 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_splH£ï$it_bytes.all.current‚ 207414272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83256023 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110960213 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416348768578288 _step‚886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429513956458496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429513956458496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429513851397120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28103279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410831160310272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7922358069760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424022905240770 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421815749112832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7698190306304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7698102284288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7845013607371733e+09 _runtime‚4434.649129303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76304243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7922175559168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83256023 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚421815766152192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48200724 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421815749112832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7698190306304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110970356 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48200727 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76304003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28103516 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424022802052936 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110960213 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429513851397120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402908984751104ö5/¬§2ò- _wandbruntime‚4434  _runtime‚4434l¿,'æ&óàM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7698868423168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76311031 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚402944704812032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424060350072140 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7698780393984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7674812614038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7922872968192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194243612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48205014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28106014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28105754 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83263344 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83263342 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚429551895874048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83263344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48205017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421853115480064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110980268 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674726465452 _runtime‚4434.923260455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410867552876544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚402944679908352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421853115480064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110970111 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7923053374464 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76310768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194233453 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110970111 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424060453260086 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83263342 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83263342 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚429551895874048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83263344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48205017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421853115480064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110980268 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674726465452 _runtime‚4434.923260455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410867552876544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚402944679908352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421853115480064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110970111 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7923053374464 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76310768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194233453 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110970111 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424060453260086 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83263342 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424060350072140 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7698780393984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7674812614038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7922872968192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194243612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48205014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28106014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28105754 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83263344 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0÷Îîƽ2õ- _wandbruntime‚4434  _runtime‚4434ζâÇ ö‡ /loss/FBt2/detection_loss‚0.5092464703031712 %loss/total_loss‚9.243453116494095 epoch‚13 % _timestamp‚1.7845013610131862e+09 0loss/FRt2/assignment_loss‚1.0441157575293536 4loss/FBt1/assignment_loss‚0.00034286780814025205 /loss/FRt1/detection_loss‚0.5287062129534431 1loss/SRqqt2/detection_loss‚0.6679838482352973 /loss/FBt1/detection_loss‚0.6772164118786653 trainer/global_step‚10949 2loss/SRqqt1/assignment_loss‚0.6832088449996632 2loss/SRqqt2/assignment_loss‚0.6290793622420762 3loss/FBt2/assignment_loss‚0.0012372702787648516 _runtime‚4434.923838436 /loss/FRt2/detection_loss‚1.5251338815123037 _step‚888 /loss/FRt1/assignment_loss‚2.075416129248345 1loss/SRqqt1/detection_loss‚0.9017660595048722ÿs*q÷Œ /loss/FRt2/detection_loss‚1.5251338815123037 _step‚888 /loss/FRt1/assignment_loss‚2.075416129248345 1loss/SRqqt1/detection_loss‚0.9017660595048722 /loss/FBt2/detection_loss‚0.5092464703031712 %loss/total_loss‚9.243453116494095 epoch‚13 % _timestamp‚1.7845013610131862e+09 0loss/FRt2/assignment_loss‚1.0441157575293536 4loss/FBt1/assignment_loss‚0.00034286780814025205 /loss/FRt1/detection_loss‚0.5287062129534431 1loss/SRqqt2/detection_loss‚0.6679838482352973 /loss/FBt1/detection_loss‚0.6772164118786653 trainer/global_step‚10949 2loss/SRqqt1/assignment_loss‚0.6832088449996632 2loss/SRqqt2/assignment_loss‚0.6290793622420762 3loss/FBt2/assignment_loss‚0.0012372702787648516 _runtime‚4434.923838436ø*Û¶‹tø‚(:é ö¨õÒ÷Ó+ disk.md0.in‚0 disk.md0.out‚ 136.76953125 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 26.22265625$ memory_percent‚99.12967681884766) disk./.usagePercent‚76.05301348088793# disk.dm-0.out‚90096.59814453125! disk.dm-1.out‚3110.2001953125( disk.nvme0c0n1.out‚68341.61474609375% proc.memory.availableMB‚ 142.59375# proc.memory.rssMB‚ 2862.66015625 cpu‚26.284994825458234 proc.cpu.threads‚17$ disk./.usageGB‚1167.603889465332 disk.dm-0.in‚ 31.43359375 disk.rbd0.out‚0& disk.nvme0n1.out‚68341.61474609375 disk.nvme1c1n1.in‚ 5.2109375 network.sent‚ 1.4619862e+07 network.recv‚ 1.289975e+06$ disk.nvme1n1.out‚25156.482421875 disk.nvme0n1.in‚ 26.22265625& disk.nvme1c1n1.out‚25156.482421875* proc.memory.percent‚17.472290992736816 disk.nvme1n1.in‚ 5.2109375 H;Lšù‚(: ö¨õÒíûÿ+ gpu.0.gpu‚77.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.044' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.01466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kà,¦2ú- _wandbruntime‚4454  _runtime‚4454ç‚2üÈû _step‚889 & lr-AdamW/pg1‚8.572759441447543e-05 & lr-AdamW/pg2‚8.572759441447543e-05 trainer/global_step‚10999 % _timestamp‚1.7845013810682707e+09 _runtime‚4454.979230245Zíá‡aüÇ _step‚889 & lr-AdamW/pg1‚8.572759441447543e-05 & lr-AdamW/pg2‚8.572759441447543ÞÞfle-05 trainer/global_step‚10999 % _timestamp‚1.7845013810682707e+09 _runtime‚4454.979230245ù!F¼2ý- _wandbruntime‚4454  _runtime‚4454~ÛV2Ý'þ×O ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404693742613504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚10999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83622071 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412651087500800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚890 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111456965 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7957319983616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423684067474432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7732040026112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48415224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431416107500544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28227314 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7707966137890 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195079036 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111467108 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚4454.980370018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28227069 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431416212561920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195089181 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418192354926928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7707879989416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76642541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83622073 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7732040026112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412650884280832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423684084513792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425900234916344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404693742613504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚10999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83622071 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412651087500800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚890 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111456965 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7957319983616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423684067474432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7732040026112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48415224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431416107500544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28227314 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7707966137890 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195079036 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111467108 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚4454.980370018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28227069 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431416212561920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195089181 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418192354926928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7707879989416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76642541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83622073 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7732040026112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412650884280832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423684084513792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425900234916344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚418229244034048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4455.255495926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423721450881024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195106414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195096255 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83629394 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111477020 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845013813443716e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195096255 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76649030 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195106414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423721450881024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111466863 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431454151977472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423721433841664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7732806164992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28229774 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425937782935548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28229516 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚404729462674432 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111466863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7957833952256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425937886123494 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7708555940860 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76649291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48419517 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10999 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7732718135808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚418229244034048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4455.255495926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423721450881024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195106414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195096255 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83629394 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111477020 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845013813443716e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195096255 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76649030 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195106414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423721450881024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111466863 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431454151977472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423721433841664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7732806164992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28229774 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425937782935548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28229516 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚404729462674432 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111466863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7957833952256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425937886123494 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7708555940860 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76649291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48419517 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10999 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7732718135808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7992225839104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433318363603968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111953717 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28350893 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48629724 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406478525379584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406478500475904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433318468665344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7766065789952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83988121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚433318363603968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48629727 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76980617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4475.310672577 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7741795613298 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83988121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76980856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111963860 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7992225839104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433318363603968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111953717 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28350893 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48629724 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406478525379584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406478500475904 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433318468665344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7766065789952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83988121 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚433318363603968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48629727 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76980617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4475.310672577 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7741795613298 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83988121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76980856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111963860 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76987627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7766743906816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433356408080896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83995442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414507136437760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420072830382688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845014016745672e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83995442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7992915900928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48634017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28353366 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433356513149440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425589752203264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7742471564854 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111963615 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414506931127808 trainer/global_step‚11049 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7766743906816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7766655877632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83995444 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425589769242624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406514195633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427815215798956 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚433356513149440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425589769242624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427815318986902 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7742385416268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195959057 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425589752203264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7766655877632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195969216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚4475.585738782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76987380 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195969216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83995444 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195959057 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚406514220536832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111973772 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48634014 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28353610 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111973772 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚895 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111963615 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157]MQì&’æM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111963615 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414506931127808 trainer/global_step‚11049 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7766743906816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7766655877632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83995444 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425589769242624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406514195633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427815215798956 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚433356513149440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425589769242624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427815318986902 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7742385416268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195959057 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425589752203264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7766655877632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195969216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚4475.585738782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76987380 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195969216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83995444 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195959057 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚406514220536832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111973772 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48634014 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28353610 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111973772 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚895 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111963615 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76987627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7766743906816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433356408080896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83995442 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414507136437760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420072830382688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845014016745672e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83995442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7992915900928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48634017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28353366 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433356513149440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425589752203264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7742471564854 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482ÿ¡:±ï2“- _wandbruntime‚4475  _runtime‚4475¶dp”‚ 0loss/FRt2/assignment_loss‚1.0055685994671841 2loss/SRqqt1/assignment_loss‚0.7138728019558148 1loss/FBt1/assignment_loss‚0.07271944429865734 /loss/SRqqt1/detection_loss‚1.10277353987506 trainer/global_step‚11049 0loss/FRt1/assignment_loss‚2.1860603620669594 .loss/FRt2/detection_loss‚1.470154352029603 0loss/FBt2/detection_loss‚0.37906321505683344 %loss/total_loss‚9.666093129573962 _step‚896 2loss/SRqqt2/assignment_loss‚0.5158017525938616 /loss/FRt1/detection_loss‚0.5563791528553536 1loss/SRqqt2/detection_loss‚0.6903745742629759 /loss/FBt1/detection_loss‚0.9710631839146739 _runtime‚4475.586918915 3loss/FBt2/assignment_loss‚0.0022621511969866505 epoch‚13 $ _timestamp‚1.784501401676123e+09ó» •‡ .loss/FRt2/detection_loss‚1.470154352029603 0loss/FBt2/detection_loss‚0.37906321505683344 %loss/total_loss‚9.666093129573962 _step‚896 2loss/SRqqt2/assignment_loss‚0.5158017525938616 /loss/FRt1/detection_loss‚0.5563791528553536 1loss/SRqqt2/detection_loss‚0.6903745742629759 /loss/FBt1/detection_loss‚0.9710631839146739 _runtime‚4475.586918915 3loss/FBt2/assignment_loss‚0.0022621511969866505 epoch‚13 $ _timestamp‚1.784501401676123e+09 0loss/FRt2/assignment_loss‚1.0055685994671841 2loss/SRqqt1/assignment_loss‚0.7138728019558148 1loss/FBt1/assignment_loss‚0.07271944429865734 /loss/SRqqt1/detection_loss‚1.10277353987506 trainer/global_step‚11049 0loss/FRt1/assignment_loss‚2.1860603620669594€ÖŽéw–‚(:ì £©õÒŽè´+) disk./.usagePercent‚76.05297447046809# disk.dm-0.out‚90117.46337890625 disk.rbd0.out‚0$ memory_percent‚99.13043975830078 proc.cpu.threads‚17 network.recv‚ 1.302822e+06( disk.nvme0c0n1.out‚68360.36865234375& disk.nvme0n1.out‚68360.36865234375( disk.nvme1c1n1.out‚25696.24462890625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2862.66015625% disk./.usageGB‚1167.6032905578613! disk.nvme0c0n1.in‚ 26.22265625 disk.nvme0n1.in‚ 26.22265625& disk.nvme1n1.out‚25696.24462890625 disk.md0.out‚ 137.94140625 disk.rbd0.in‚0% proc.memory.availableMB‚ 142.46875 cpu‚26.280923817482 network.sent‚ 1.4773072e+07 disk.nvme1c1n1.in‚ 5.2109375 disk.nvme1n1.in‚ 5.2109375 disk.md0.in‚0 disk.dm-0.in‚ 31.43359375" disk.dm-1.out‚3645.33935546875* proc.memory.percent‚17.472290992736816üó—‚(:„ £©õÒ¹ôË+ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚143.664' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚47.888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õƒI2˜- _wandbruntime‚4495  _runtime‚4495›ï÷ș & lr-AdamW/pg1‚8.542976172663713e-05 & lr-AdamW/pg2‚8.542976172663713e-05 trainer/global_step‚11099 % _timestamp‚1.7845014213839128e+09 _runtime‚4495.294841963 _step‚8975gÕÍšÇ trainer/global_step‚11099 % _timestamp‚1.7845014213839128e+09 _runtime‚4495.294841963 _step‚897 & lr-AdamW/pg1‚8.542976172663713e-05 & lr-AdamW/pg2‚8.542976172663713e-05v2›- _wandbruntime‚4495  _runtime‚4495ÜZni?#œÖO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435220619707392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28474028 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196804640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112450469 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚427420704197632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7800003531776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7775538940232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 $ _timestamp‚1.784501421384872e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7800003531776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112460612 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427420721236992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48844227 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427420721236992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421879561702928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28473792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416290409062400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435220619707392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7775625088706 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚435220724768768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8027125820416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77318016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84354171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435220724768768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48844224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429655100643160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7799915509760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84354173 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77318255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416290201648128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8026943309824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84354173 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7799915509760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84354171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408263258338304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427420704197632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112450469 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408263283241984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196814785 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _step‚898 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsM"æÞÉonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421879578742288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚11099 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚4495.295729834¬LJÆá'ÛO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421879578742288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚11099 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚4495.295729834 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435220619707392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28474028 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196804640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112450469 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚427420704197632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7800003531776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7775538940232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 $ _timestamp‚1.784501421384872e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7800003531776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112460612 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427420721236992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48844227 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427420721236992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421879561702928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28473792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416290409062400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435220619707392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7775625088706 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚435220724768768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8027125820416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77318016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84354171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435220724768768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48844224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429655100643160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7799915509760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84354173 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77318255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416290201648128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8026943309824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84354173 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7799915509760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84354171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408263258338304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 202022912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427420704197632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112450469 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408263283241984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196814785 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _step‚898 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239‚ï±Ü2ž- _wandbruntime‚4495  _runtime‚4495n—ωç&ŸáM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48848517 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429692648662364 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112460367 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427458087604224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚435258769252864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 _step‚899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427458087604224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84361494 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77324967 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416326590829056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48848514 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28476450 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196832018 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421916450810048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429692751850310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4495.890427624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84361492 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8027824031232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408298978399232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7800681648640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77324723 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421916433770688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7776301040262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7776214891676 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196821859 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 % _timestamp‚1.7845014219793036e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196832018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84361494 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84361492 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408298953495552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112470524 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7800593619456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8027637333504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48848514 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28476450 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196832018 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421916450810048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429692751850310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4495.890427624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84361492 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8027824031232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408298978399232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7800681648640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77324723 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421916433770688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7776301040262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7776214891676 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196821859 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 % _timestamp‚1.7845014219793036e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196832018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84361494 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84361492 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408298953495552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112470524 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7800593619456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8027637333504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427458087604224ƒ>!2¡- _wandbruntime‚4495  _runtime‚4495•-l ¢„ .loss/FRt2/detection_loss‚1.567085354530203 0loss/SRqqt1/detection_loss‚1.013343948651082 /loss/FBt1/detection_loss‚0.5709072877431199 %loss/total_loss‚9.102419043170885 trainer/global_step‚11099 _runtime‚4495.891175125 0loss/FRt1/assignment_loss‚2.0301482648476488 0loss/FRt2/assignment_loss‚0.9604376156245713 2loss/SRqqt1/assignment_loss‚0.5669361689650043 3loss/FBt2/assignment_loss‚0.0006040936030951484 epoch‚13 % _timestamp‚1.7845014219804142e+09 1loss/SRqqt2/assignment_loss‚0.617315344625061 /loss/FRt1/detection_loss‚0.6084507275738695 /loss/FBt2/detection_loss‚0.5117538966373295 3loss/FBt1/assignment_loss‚0.0004974792961474707 1loss/SRqqt2/detection_loss‚0.6549388610737531 _step‚900øZ’7 £‰ .loss/FRt2/detection_loss‚1.567085354530203 0loss/SRqqt1/detection_loss‚1.013343948651082 /loss/FBt1/detection_loss‚0.5709072877431199 %loss/total_loss‚9.102419043170885 trainer/global_step‚11099 _runtime‚4495.891175125 0loss/FRt1/assignment_loss‚2.0301482648476488 0loss/FRt2/assignment_loss‚0.9604376156245713 2loss/SRqqt1/assignment_loss‚0.5669361689650043 3loss/FBt2/assignment_loss‚0.0006040936030951484 epoch‚13 % _timestamp‚1.7845014219804142e+09 1loss/SR˲Êgqqt2/assignment_loss‚0.617315344625061 /loss/FRt1/detection_loss‚0.6084507275738695 /loss/FBt2/detection_loss‚0.5117538966373295 3loss/FBt1/assignment_loss‚0.0004974792961474707 1loss/SRqqt2/detection_loss‚0.6549388610737531 _step‚900„ùtWÉy¤‚(:î ²©õÒþž+$ memory_percent‚99.13232326507568 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.in‚0# proc.memory.rssMB‚ 2862.66015625* proc.memory.percent‚17.472290992736816 cpu‚26.279549165837803 network.recv‚ 1.306846e+06% disk./.usageGB‚1167.6033744812012! disk.nvme0c0n1.in‚ 26.22265625 disk.nvme1n1.in‚ 5.2109375 disk.dm-0.in‚ 31.43359375 proc.cpu.threads‚17) disk./.usagePercent‚76.05297993689635 disk.nvme0n1.in‚ 26.22265625( disk.nvme1c1n1.out‚25696.65576171875" disk.dm-0.out‚90118.6005859375" disk.dm-1.out‚3645.38232421875( proc.memory.availableMB‚ 142.16015625 network.sent‚ 1.4835291e+07' disk.nvme0c0n1.out‚68361.6865234375% disk.nvme0n1.out‚68361.6865234375 disk.nvme1c1n1.in‚ 5.2109375& disk.nvme1n1.out‚25696.65576171875 disk.md0.out‚ 138.1953125 disk.dm-1.in‚0Ÿ’y/š¥‚(: ²©õÒ–¾+ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚118.013' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.33766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oC™¦‚(:Ž Á©õÒ¡™—+ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚170.932' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚56.97733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’û¦•v§‚(:ë Á©õÒœ½£+ disk.nvme1n1.in‚ 5.2109375 disk.md0.out‚ 138.76171875 disk.rbd0.out‚0! disk.dm-0.out‚90123.814453125 cpu‚26.27823613980231 disk.nvme1c1n1.in‚ 5.2109375) disk./.usagePercent‚76.05285967547465" disk.dm-1.out‚3645.82763671875' disk.nvme0c0n1.out‚68367.0029296875 disk.nvme0n1.in‚ 26.22265625& disk.nvme1n1.out‚25698.20361328125 disk.md0.in‚0# proc.memory.rssMB‚ 2862.66015625% disk./.usageGB‚1167.6015281677246! disk.nvme0c0n1.in‚ 26.22265625 disk.dm-0.in‚ 31.43359375$ memory_percent‚99.13244247436523& proc.memory.availableMB‚ 142.140625* proc.memory.percent‚17.472290992736816 proc.cpu.threads‚17% disk.nvme0n1.out‚68367.0029296875( disk.nvme1c1n1.out‚25698.20361328125 network.sent‚ 1.4897562e+07 network.recv‚ 1.311223e+06 disk.dm-1.in‚0 disk.rbd0.in‚0ºîóÀ2¨- _wandbruntime‚4515  _runtime‚4515ó¥Ë™È©Â _step‚901 & lr-AdamW/pg1‚8.528001408015521e-05 & lr-AdamW/pg2‚8.528001408015521e-05 trainer/global_step‚11149 % _timestamp‚1.7845014416836498e+09 _runtime‚4515.595375809ã¥øT껂 & lr-AdamW/pg1‚8.528001408015521e-05 & lr-AdamW/pg2‚8.528001408015521e-05 trainer/global_step‚11149 % _timestamp‚1.7845014416836498e+09 _runtime‚4515.595375809 _step‚901…ΙjŠ2«- _wandbruntime‚4515  _runtime‚4515Sy¾+Ý'¬×O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7833853251584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429289022559232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429289022559232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28596586 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429289039598592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49058727 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410048041104384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49058724 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197677587 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437122875810816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84720221 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410048016200704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429289039598592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84720223 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28596825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112957364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437122980872192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303Jc´6â'­ÜO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437122875810816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84720221 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410048016200704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429289039598592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84720223 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28596825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112957364 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437122980872192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7833853251584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429289022559232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429289022559232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28596586 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429289039598592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49058727 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410048041104384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49058724 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197677587 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7810130515670 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7834619390464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7834531361280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77662080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28599316 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431570184713718 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8062792099328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49063014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7834531361280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418146327245312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84727542 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28599066 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437161025356288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197694820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77662333 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚11149 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49063017 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423760037158688 _runtime‚4515.869058738 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84727544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423760054198048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112957119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429326405965824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚429326388926464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84727542 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112957119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429326388926464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8062615887360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437161025356288 H:DeviceStatsMonitor.onmåÑT_train_batch_end/allocation.all.freed‚ 197684661 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418146528360960!jÒì&°æM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84727544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423760054198048 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112957119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429326405965824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚429326388926464 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84727542 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112957119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429326388926464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8062615887360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437161025356288 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197684661 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418146528360960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7810130515670 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7834619390464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7834531361280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77662080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28599316 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431570184713718 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8062792099328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49063014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7834531361280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418146327245312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84727542 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28599066 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437161025356288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197694820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77662333 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚11149 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49063017 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423760037158688 _runtime‚4515.869058738 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159‡Êpóv2±- _wandbruntime‚4515  _runtime‚4515ýÐib² 0loss/FRt2/assignment_loss‚1.0757982074785575 /loss/FRt1/detection_loss‚0.6100274945187877 /loss/FBt1/detection_loss‚0.7689008919092325 %loss/total_loss‚9.396126505364581 epoch‚13 _step‚904 2loss/SRqqt1/assignment_loss‚0.6659457984300022 2loss/FBt2/assignment_loss‚0.005600705368581194 1loss/SRqqt1/detection_loss‚1.0402147596728781 /loss/FBt2/detection_loss‚0.4982741988176291 trainer/global_step‚11149 _runtime‚ 4515.86972705 0loss/FRt1/assignment_loss‚2.0044560563124594 2loss/SRqqt2/assignment_loss‚0.5803945467527654 1loss/FBt1/assignment_loss‚0.10978549525861817 0loss/SRqqt2/detection_loss‚0.603513478107944 .loss/FRt2/detection_loss‚1.433214872737127 % _timestamp‚1.7845014419590378e+09Ù‚…Î ³† .loss/FRt2/detection_loss‚1.433214872737127 % _timestamp‚1.7845014419590378e+09 0loss/FRt2/assignment_loss‚1.0757982074785575 /loss/FRt1/detection_loss‚0.6100274945187877 /loss/FBt1/detection_loss‚0.7689008919092325 %loss/total_loss‚9.396126505364581 epoch‚13 _step‚904 2loss/SRqqt1/assignment_loss‚0.6659457984300022 2loss/FBt2/assignment_loss‚0.005600705368581194 1loss/SRqqt1/detection_loss‚1.0402147596728781 /loss/FBt2/detection_loss‚0.4982741988176291 trainer/global_step‚11149 _runtime‚ 4515.86972705 0loss/FRt1/assignment_loss‚2.0044560563124594 2loss/SRqqt2/assignment_loss‚0.5803945467527654 1loss/FBt1/assignment_loss‚0.10978549525861817 0loss/SRqqt2/detection_loss‚0.603513478107944ˆ¶yª´‚(:„ ЩõÒîü•+ gpu.0.gpu‚52.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚117.378' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ3ærµ‚(:ç ЩõÒÜ Ú+ cpu‚26.276927067029824 network.sent‚ 1.4915278e+07 network.recv‚ 1.314558e+06 disk.dm-0.in‚ 36.44921875 disk.dm-0.out‚ 90124.90625" disk.dm-1.out‚3645.89404296875& disk.nvme1n1.out‚25698.70849609375) disk./.usagePercent‚76.05291806686742$ disk./.usageGB‚1167.602424621582 disk.md0.out‚ 138.99609375! disk.nvme0c0n1.in‚ 31.23828125 disk.nvme0n1.in‚ 31.23828125 disk.nvme1c1n1.in‚ 5.2109375( disk.nvme1c1n1.out‚25698.70849609375 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚68368.166015625$ disk.nvme0n1.out‚68368.166015625 disk.nvme1n1.in‚ 5.2109375$ memory_percent‚99.13289546966553( proc.memory.availableMB‚ 142.06640625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2862.66015625* proc.memory.percent‚17.472290992736816ÎÒkº2¶- _wandbruntime‚4535  _runtime‚4535SåóÇ·Á & lr-AdamW/pg1‚8.512971542702682e-05 & lr-AdamW/pg2‚8.512971542702682e-05 trainer/global_step‚11199 $ _timestamp‚1.784501461969307e+09 _runtime‚4535.880277099 _step‚905ÃP3OÌ¸Æ & lr-AdamW/pg1‚8.512971542702682e-05 & lr-AdamW/pg2‚8.512971542702682e-05 trainer/global_step‚11199 $ _timestamp‚1.784501461969307e+09 _runtime‚4535.880277099 _step‚905‰+~æ2¹- _wandbruntime‚4535  _runtime‚4535qßb6Ü'ºÖO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113454116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7867879015424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433409966369976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439025131914240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8096806756864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411832798966784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198530244 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113454116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77993509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425566768478928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198540389 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439025236975616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425566785518288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431157340920832 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439025236975616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431157340920832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113443973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚906 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49273224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845014619703798e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28720045 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚ 4535.88146616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85086273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7867790993408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592>Ž0RL»ÛO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198530244 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113454116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77993509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425566768478928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198540389 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439025236975616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425566785518288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431157340920832 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439025236975616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431157340920832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113443973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚906 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49273224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845014619703798e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28720045 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚ 4535.88146616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85086273 WEDeviceStatsMonitor. È¥•on_train_batch_start/active_bytes.small_pool.freed‚ 7867790993408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113454116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7867879015424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433409966369976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439025131914240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8096806756864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411832798966784ŠÇûíè2¼- _wandbruntime‚4536  _runtime‚4536£Ô.Êç&½áM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85093594 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 % _timestamp‚1.7845014622491233e+09 _step‚907 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8097500352512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85093594 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411868494124032 trainer/global_step‚11199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198557622 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7868469103104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚439063281459712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425603657586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7843959991078 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113464028 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85093592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7868469103104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431194707288064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49277514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28722399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419966179077120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78000160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198557622 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113453871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77999913 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7868557132288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113453871 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198547463 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8097684953088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚4536.160519041 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85093592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433447514389180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425603640546688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431194707288064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113464028 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433447617577126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28722643s ì&¾æM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411868469220352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7843873842492 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198557622 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113453871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77999913 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7868557132288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113453871 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198547463 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8097684953088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚4536.160519041 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85093592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433447514389180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425603640546688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431194707288064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113464028 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433447617577126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28722643 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85093594 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 202022912 % _timestamp‚1.7845014622491233e+09 _step‚907 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8097500352512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85093594 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411868494124032 trainer/global_step‚11199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198557622 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7868469103104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚439063281459712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425603657586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7843959991078 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113464028 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85093592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7868469103104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431194707288064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49277514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28722399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419966179077120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78000160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 )jet/accuracy_2_of_2‚0.155418261885643 'particle/f_score‚0.6594851613044739 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6975486278533936 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15872299671173096 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6494123935699463 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48956334590911865 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7110140919685364 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14791180193424225 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 )jet/accuracy_2_of_3‚0.565518856048584 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 trainer/global_step‚11213 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16326719522476196 *jet/accuracy_1_of_2‚0.7980437278747559 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.043638501316308975 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8659511804580688 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5692843198776245 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7382241487503052 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9860780835151672 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.815733790397644 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15536649525165558 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8440476655960083 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 jet/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4831688404083252 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17901943624019623 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04328081011772156 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04387960955500603 /particle/accuracy_4_of_4‚0.9463527202606201 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7315784692764282 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15000000596046448 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6856235861778259 )jet/accuracy_1_of_3‚0.932981014251709 +jet/accuracy_3_of_4‚0.09395195543766022 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9150067567825317 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8315789103507996 % _timestamp‚1.7845014839735417e+09 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14443883299827576 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03157131001353264 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6898216009140015 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1726558804512024 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03215215727686882 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.031089382246136665 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45857465267181396 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6395285725593567 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04732692986726761 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98301762342453 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.047996725887060165 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.740890383720398 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1859719455242157 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.031089382246136665 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29913270473480225 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6506815552711487 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995894193649292 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0070297857746481895 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21844477951526642 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5695049166679382 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5869772434234619 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6124998927116394 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16051353514194489 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8928571343421936 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18591973185539246 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14134755730628967 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016445696353912354 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15602971613407135 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17364157736301422 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 particle/accuracy_1_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939901232719421 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30848103761672974 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7314651608467102 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,jet/accuracy_4_of_4‚0.018498167395591736 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7603669166564941 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8679835200309753 9"Purit³Nãüy/0FRt*SRqqt2FBt/SRqqt_purity‚0.2142857164144516 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7445908784866333 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983035326004028 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8722666501998901 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939199090003967 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_3‚1 /particle/accuracy_3_of_4‚0.9981684684753418 _runtime‚4557.884937671 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.616636335849762 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 epoch‚13 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7018723487854004 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49733662605285645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7664817571640015 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9730905890464783 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8388583660125732 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9323137402534485 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6388648748397827 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 +particle/specificity‚0.9081805348396301 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15000000596046448 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6551825404167175 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988133311271667 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9721560478210449 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916588664054871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37037035822868347 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 _step‚909 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24114803969860077 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45757824182510376 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17122668027877808 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970504999160767 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5511643290519714 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8176558613777161 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18535010516643524 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.686547577381134 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6569622159004211 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_3_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871968030929565 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2603038549423218 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6938144564628601 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16397161781787872 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38774609565734863 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6821743845939636 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1658017486333847 +jet/accuracy_3_of_3‚0.07531159371137619 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7636751532554626 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952704906463623 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14468860626220703 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7713049054145813 jet/accuracy_1_of_5‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8315789103507996 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22259871661663055 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.36407214403152466 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7478436231613159 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5467099547386169 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6171467900276184 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_4_of_5‚1 (loss/val_total_loss‚8.91418774490763 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 (particle/accuracy‚0.8542905449867249 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4765149652957916 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980934858322144 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8679835200309753 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39772728085517883 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949344396591187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9935052990913391 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1726558804512024 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8384614586830139 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666865348816 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1807357519865036 +particle/sensitivity‚0.6571987867355347 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6551825404167175 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6153135895729065 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20070791244506836 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6623759865760803 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1323770135641098 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_2‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983035326004028 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014046786352992058 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7710487842559814 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8141642212867737 - Purity/2FRt*SRqqt1FBt/FRt_purity‚0.515625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2142857164144516 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38490018248558044 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7331714034080505Qœ=Æ–{ ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1807357519865036 +particle/sensitivity‚0.6571987867355347 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6551825404167175 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6153135895729065 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20070791244506836 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6623759865760803 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1323770135641098 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_2‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983035326004028 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014046786352992058 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7710487842559814 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8141642212867737 - Purity/2FRt*SRqqt1FBt/FRt_purity‚0.515625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2142857164144516 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38490018248558044 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7331714034080505 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17249451577663422 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7377904057502747 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9842707514762878 *jet/accuracy_1_of_1‚0.3158269226551056 particle/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 )jet/accuracy_2_of_2‚0.155418261885643 'particle/f_score‚0.6594851613044739 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6975486278533936 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15872299671173096 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6494123935699463 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48956334590911865 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7110140919685364 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14791180193424225 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 )jet/accuracy_2_of_3‚0.565518856048584 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.925000011920929 trainer/global_step‚11213 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16326719522476196 *jet/accuracy_1_of_2‚0.7980437278747559 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.043638501316308975 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8659511804580688 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5692843198776245 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7382241487503052 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9860780835151672 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.815733790397644 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15536649525165558 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8440476655960083 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 jet/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4831688404083252 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17901943624019623 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04328081011772156 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04387960955500603 /particle/accuracy_4_of_4‚0.9463527202606201 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7315784692764282 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15000000596046448 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6856235861778259 )jet/accuracy_1_of_3‚0.932981014251709 +jet/accuracy_3_of_4‚0.09395195543766022 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9150067567825317 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8315789103507996 % _timestamp‚1.7845014839735417e+09 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14443883299827576 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03157131001353264 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6898216009140015 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1726558804512024 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03215215727686882 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.031089382246136665 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45857465267181396 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6395285725593567 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04732692986726761 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98301762342453 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.047996725887060165 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.740890383720398 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1859719455242157 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.031089382246136665 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29913270473480225 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6506815552711487 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995894193649292 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0070297857746481895 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21844477951526642 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5695049166679382 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5869772434234619 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6124998927116394 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16051353514194489 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8928571343421936 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18591973185539246 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14134755730628967 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016445696353912354 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15602971613407135 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17364157736301422 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 particle/accuracy_1_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939901232719421 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30848103761672974 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7314651608467102 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,jet/accuracy_4_of_4‚0.018498167395591736 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7603669166564941 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8679835200309753 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2142857164144516 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7445908784866333 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983035326004028 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8722666501998901 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939199090003967 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_3‚1 /particle/accuracy_3_of_4‚0.9981684684753418 _runtime‚4557.884937671 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.616636335849762 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 epoch‚13 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7018723487854004 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49733662605285645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7664817571640015 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9730905890464783 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8388583660125732 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9323137402534485 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6388648748397827 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 +particle/specificity‚0.9081805348396301 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15000000596046448 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6551825404167175 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988133311271667 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9721560478210449 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916588664054871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37037035822868347 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 _step‚909 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24114803969860077 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45757824182510376 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17122668027877808 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970504999160767 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5511643290519714 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.8176558613777161 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18535010516643524 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.686547577381134 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6569622159004211 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_3_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871968030929565 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2603038549423218 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6938144564628601 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16397161781787872 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38774609565734863 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6821743845939636 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1658017486333847 +jet/accuracy_3_of_3‚0.07531159371137619 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7636751532554626 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9952704906463623 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14468860626220703 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7713049054145813 jet/accuracy_1_of_5‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8315789103507996 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22259871661663055 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.36407214403152466 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7478436231613159 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5467099547386169 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6171467900276184 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_4_of_5‚1 (loss/val_total_loss‚8.91418774490763 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 (particle/accuracy‚0.8542905449867249 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4765149652957916 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980934858322144 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8679835200309753 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39772728085517883 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949344396591187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9935052990913391 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1726558804512024 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8384614586830139 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666865348816BàssÇ‚(:è î©õÒЫ+! disk.dm-0.out‚90129.064453125! disk.nvme0c0n1.in‚ 31.98828125 disk.nvme1c1n1.in‚13.34375 disk.md0.out‚ 139.7109375$ memory_percent‚98.82452487945557) proc.memory.percent‚17.48521327972412 network.sent‚ 1.5026023e+07) disk./.usagePercent‚76.03751888998828$ disk./.usageGB‚1167.366008758545" disk.dm-1.out‚3709.96240234375 disk.rbd0.in‚0( disk.nvme1c1n1.out‚25763.96728515625 disk.md0.in‚0( proc.memory.availableMB‚ 192.58984375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚68372.6943359375# proc.memory.rssMB‚ 2864.77734375 cpu‚26.28495499300273 proc.cpu.threads‚17 disk.dm-0.in‚ 45.33203125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 31.98828125% disk.nvme0n1.out‚68372.6943359375 disk.nvme1n1.in‚13.34375& disk.nvme1n1.out‚25763.96728515625 network.recv‚ 1.322212e+06§áLšÈ‚(: î©õÒµˆ¼+ gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚116.815' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.93833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U•Žu2É- _wandbruntime‚4572  _runtime‚4572tKÇÊÁ & lr-AdamW/pg1‚8.497886817458997e-05 & lr-AdamW/pg2‚8.497886817458997e-05 trainer/global_step‚11249 $ _timestamp‚1.784501498637231e+09 _runtime‚4572.548087407 _step‚910뽎iÌËÆ _step‚910 & lr-AdamW/pg1‚8.497886817458997e-05 & lr-AdamW/pg2‚8.497886817458997e-05 trainer/global_step‚11249 $ _timestamp‚1.784501498637231e+09 _runtime‚4572.548087407ŽÉ %ª2Ì- _wandbruntime‚4572  _runtime‚4572÷-p›%Í×O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78508580 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85689789 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441920576991232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7922237074432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433998322877440 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199797780 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114107991 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441920471929856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78508846 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422695212225024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49581807 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7922149052416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7922149052416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚11249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚428378878374640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85689791 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28926776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441920471929856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114107991 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422695423833600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436276282709302 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114118134 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7922237074432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytšË콸 es.all.allocated‚441920576991232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433998322877440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428378861335280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845014986378353e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414542955189248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114118134 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚4572.549301969 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85689791 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7897421374022 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49581804 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7897507522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433998339916800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28927039 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433998339916800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720õð&â'ÎÜO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433998339916800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28927039 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433998339916800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78508580 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85689789 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441920576991232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7922237074432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433998322877440 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199797780 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114107991 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441920471929856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78508846 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422695212225024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49581807 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7922149052416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7922149052416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚11249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚428378878374640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85689791 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28926776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441920471929856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114107991 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422695423833600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436276282709302 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114118134 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7922237074432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚441920576991232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433998322877440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428378861335280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845014986378353e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414542955189248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114118134 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚4572.549301969 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85689791 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7897421374022 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49581804 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7897507522496½{¼à2Ï- _wandbruntime‚4572  _runtime‚4572 ªç&ÐáM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199825158 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85697112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7922827162112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚441958621475328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49586097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚414578675250176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428415750442400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85697112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85697110 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8153140314624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845014989062061e+09 _step‚912 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114128046 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7922827162112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434035706284032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7922915191296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚441958621475328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚436313830728506 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434035689244672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434035689244672 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78515379 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28929532 _runtime‚4572.817309569 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434035706284032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422731610254848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85697110 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49586094 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114128046 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚436313933916452 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199825158 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8152959908352 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚414578650346496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78515629 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114117889 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816t¥Ÿ¹*$ÑæM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434035689244672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434035689244672 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78515379 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28929532 _runtime‚4572.817309569 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434035706284032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422731610254848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85697110 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49586094 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114128046 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚436313933916452 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199825158 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8152959908352 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚414578650346496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78515629 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114117889 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199825158 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85697112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7922827162112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚441958621475328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49586097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚414578675250176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428415750442400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85697112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85697110 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8153140314624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845014989062061e+09 _step‚912 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114128046 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7922827162112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434035706284032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7922915191296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚441958621475328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068çR6dÂ544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚436313830728506 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152BSO2Ò- _wandbruntime‚4572  _runtime‚4572.× Óƒ %loss/total_loss‚9.315069837966323 epoch‚14 0loss/FRt2/assignment_loss‚0.9640863256191644 2loss/SRqqt1/assignment_loss‚0.7816379262635914 2loss/SRqqt2/assignment_loss‚0.6239472993614683 3loss/FBt1/assignment_loss‚0.0009417863309084039 /loss/FRt1/detection_loss‚0.5994084225270893 .loss/FRt2/detection_loss‚1.428835640830083 .loss/FBt2/detection_loss‚0.573769935950016 _step‚913 1loss/SRqqt1/detection_loss‚0.9453419272262508 1loss/SRqqt2/detection_loss‚0.6337252211599811 /loss/FBt1/detection_loss‚0.6863038258494011 /loss/FRt1/assignment_loss‚2.061983457524726 2loss/FBt2/assignment_loss‚0.015088069323642938 trainer/global_step‚11249 % _timestamp‚1.7845014989072556e+09 _runtime‚4572.817911239eŒëìÔˆ 1loss/SRqqt1/detection_loss‚0.9453419272262508 1loss/SRqqt2/detection_loss‚0.6337252211599811 /loss/FBt1/detection_loss‚0.6863038258494011 /loss/FRt1/assignment_loss‚2.061983457524726 2loss/FBt2/assignment_loss‚0.015088069323642938 trainer/global_step‚11249 % _timestamp‚1.7845014989072556e+09 _runtime‚4572.817911239 %loss/total_loss‚9.315069837966323 epoch‚14 0loss/FRt2/assignment_loss‚0.9640863256191644 2loss/SRqqt1/assignment_loss‚0.7816379262635914 2loss/SRqqt2/assignment_loss‚0.6239472993614683 3loss/FBt1/assignment_loss‚0.0009417863309084039 /loss/FRt1/detection_loss‚0.5994084225270893 .loss/FRt2/detection_loss‚1.428835640830083 .loss/FBt2/detection_loss‚0.573769935950016 _step‚913‘«TØÕ‚(:‚ ý©õÒéœÄ+ gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.31' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.77 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ÎrDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7956174816256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424514897630720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86055839 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚915 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86055841 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200670727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚438153818760544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78846607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430222464723280 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚435866641239040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443822833094656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443822833094656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49796304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443822728033280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435866658278400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438153715572710 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7956086794240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7931250849430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435866641239040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114614886 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7956086794240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86055841 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114604743 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443822728033280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚11299 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49796304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443822728033280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435866658278400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438153715572710 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7956086794240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7931250849430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435866641239040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114614886 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7956086794240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86055841 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114604743 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443822728033280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚11299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7956174816256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424514897630720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86055839 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚915 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86055841 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200670727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330“þ†O52ß- _wandbruntime‚4592  _runtime‚4592ü–ÀàáM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114614641 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚435904024645632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200687960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430259353830400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443860877578752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435904024645632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8187891277824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29052981 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845015186441352e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78853322 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443860877578752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚435904007606272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114614641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49800594 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424551299486720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7956764903936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416363408208896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29052728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7932012949460 _step‚916 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86063160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7956852933120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200687960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200677801 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4592.555182259 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8188061198336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚438191366779860 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7931926800874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86063162 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416363433112576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7956852933120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443860772510208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114624798 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424551494310912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443860877578752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435904024645632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8187891277824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29052981 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845015186441352e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78853322 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443860877578752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚435904007606272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114614641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49800594 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424551299486720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7956764903936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416363408208896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29052728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7932012949460 _step‚916 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86063160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7956852933120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200687960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200677801 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4592.555182259 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8188061198336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚438191366779860 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7931926800874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86063162 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416363433112576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7956852933120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443860772510208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114624798 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424551494310912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚435904024645632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0”º²92â- _wandbruntime‚4592  _runtime‚4592„SÑã‚ /loss/FRt2/detection_loss‚1.3882028412601015 _runtime‚ 4592.55576398 2loss/SRqqt1/assignment_loss‚0.6125545253536787 1loss/SRqqt1/detection_loss‚0.9834845789036778 epoch‚14 % _timestamp‚1.7845015186451287e+09 0loss/FRt2/assignment_loss‚0.9650519339543469 2loss/FBt1/assignment_loss‚0.029228674978596558 /loss/FRt1/detection_loss‚0.5124596561919728 %loss/total_loss‚9.320543176748071 trainer/global_step‚11299 2loss/SRqqt2/assignment_loss‚0.5346341867273858 0loss/FBt2/assignment_loss‚0.0084133872751391 1loss/SRqqt2/detection_loss‚0.5733437592354063 /loss/FBt1/detection_loss‚0.9073018425144255 /loss/FBt2/detection_loss‚0.5669420799097189 _step‚917 0loss/FRt1/assignment_loss‚2.2389257104436227¯ò™E ä‡ %loss/total_loss‚9.320543176748071 trainer/global_step‚11299 2loss/SRqqt2/assignment_loss‚0.5346341867273858 0loss/FBt2/assignment_loss‚0.0084133872751391 1loss/SRqqt2/detection_loss‚0.5733437592354063 /loss/FBt1/detection_loss‚0.9073018425144255 /loss/FBt2/detection_loss‚0.5669420799097189 _step‚917 0loss/FRt1/assignment_loss‚2.2389257104436227 /loss/FRt2/detection_loss‚1.3882028412601015 _runtime‚ 4592.55576398 2loss/SRqqt1/assignment_loss‚0.6125545253536787 1loss/SRqqt1/detection_loss‚0.9834845789036778 epoch‚14 % _timestamp‚1.7845015186451287e+09 0loss/FRt2/assignment_loss‚0.9650519339543469 2loss/FBt1/assignment_loss‚0.029228674978596558 /loss/FRt1/detection_loss‚0.5124596561919728•û¡šxå‚(:í ›ªõÒ­ÆÓ+$ disk./.usageGB‚1167.127944946289 disk.nvme1n1.in‚13.34375 disk.dm-0.in‚ 45.33203125 disk.dm-1.in‚0" disk.dm-1.out‚4393.76123046875! disk.nvme0c0n1.in‚ 31.98828125( disk.nvme1c1n1.out‚26452.76123046875# proc.memory.rssMB‚ 2864.77734375 network.sent‚ 1.515012e+07 network.recv‚ 1.334148e+06# disk.dm-0.out‚90155.44189453125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚13.34375$ memory_percent‚98.82495403289795( proc.memory.availableMB‚ 192.51953125) proc.memory.percent‚17.48521327972412 cpu‚26.281145984547408 proc.cpu.threads‚17) disk./.usagePercent‚76.02201237233605& disk.nvme1n1.out‚26452.76123046875 disk.md0.in‚0 disk.md0.out‚ 140.67578125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚68396.16064453125 disk.nvme0n1.in‚ 31.98828125& disk.nvme0n1.out‚68396.16064453125Å¥±×šæ‚(: ›ªõÒ²®¨, gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.405' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.80166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A 02ç- _wandbruntime‚4612  _runtime‚4612î£ñéÈè _step‚918 & lr-AdamW/pg1‚8.467553754503881e-05 & lr-AdamW/pg2‚8.467553754503881e-05 trainer/global_step‚11349 % _timestamp‚1.7845015381704943e+09 _runtime‚4612.082405109ÿˆ‡gÍéÇ & lr-AdamW/pg1‚8.467553754503881e-05 & lr-AdamW/pg2‚8.467553754503881e-05 trainer/global_step‚11349 % _timestamp‚1.7845015381704943e+09 _runtime‚4612.082405109 _step‚918–Ãí@N2ê- _wandbruntime‚4612  _runtime‚4612ßœµÝ'ë×O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7965166473312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7990112558080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437734976640000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50010807 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418112495817728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚440031148436118 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7845015381712027e+09 _runtime‚4612.082465399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79184223 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86421889 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445725089198080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115101495 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚437734959600640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201533529 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8222359280128 trainer/global_step‚11349 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8222162089472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚437734959600640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115111638 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445725089198080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437734976640000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197190656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115101495 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29173680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426334855097856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201523384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445724984136704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432066068111280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _step‚919 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86421891 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86421891 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115111638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50010804 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7990024536064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418112470914048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440031251623952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86421889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222094336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7990112558080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79184487 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445724984136704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0iM2‚3ìÜO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201523384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445724984136704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432066068111280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _step‚919 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86421891 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86421891 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115111638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50010804 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7990024536064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418112470914048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440031251623952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86421889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222094336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7990112558080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79184487 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445724984136704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7965166473312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7990112558080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Wó‡Ê¯JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437734976640000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50010807 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418112495817728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚440031148436118 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7845015381712027e+09 _runtime‚4612.082465399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79184223 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86421889 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445725089198080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115101495 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚437734959600640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201533529 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8222359280128 trainer/global_step‚11349 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8222162089472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚437734959600640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115111638 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445725089198080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437734976640000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197190656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115101495 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29173680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426334855097856—8,*T2í- _wandbruntime‚4612  _runtime‚4612ÒŠÍ9ç&îáM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50015094 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚11349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50015097 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7965756276282 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201540603 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115111393 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8222860440576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86429210 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418148166071296 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚432102940179040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845015384415896e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201550762 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437772325967872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437772343007232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8223045041152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440068799643268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426371026511872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445763133682176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86429212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445763028613632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7990702645760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201550762 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79191003 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201540603 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990790674944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7990790674944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437772343007232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115121550 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 _step‚920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418148190974976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29176176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E´ì&ïæM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115121550 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 _step‚920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418148190974976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29176176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚11349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50015097 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7965756276282 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201540603 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115111393 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8222860440576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86429210 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418148166071296 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚432102940179040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845015384415896e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201550762 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437772325967872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437772343007232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8223045041152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440068799643268 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426371026511872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445763133682176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86429212 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚445763028613632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7990702645760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201550762 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79191003 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201540603 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990790674944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7990790674944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437772343007232˜ÇÙ²2ð- _wandbruntime‚4612  _runtime‚4612DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202386186 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447627240240128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439603295001600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428154325438976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433909688538640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚447627345301504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8023962277888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202396331 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8024050299904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7998909800246 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86787941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419897253680128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8257096662528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115608390 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439603277962240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 trainer/global_step‚11399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8024050299904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447627240240128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115598247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86787941 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441908684487360 _step‚923$nYá'úÛO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8024050299904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447627240240128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115598247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86787941 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441908684487360 _step‚923 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433909671499280 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86787939 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50225304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441908581299526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79522744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439603295001600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115608390 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202386186 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447627240240128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439603295001600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428154325438976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433909688538640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚447627345301504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8023962277888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202396331 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8024050299904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7998909800246 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86787941 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419897253680128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8257096662528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115608390 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439603277962240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 trainer/global_step‚11399›R}U2û- _wandbruntime‚4632  _runtime‚4632à [Læ&üàM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86795260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202403405 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447665389785600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419932948837376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8024640387584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86795260 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433946543567040 _runtime‚4632.286277793 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202403405 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79529691 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29300094 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419932923933696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50229594 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29299844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441946232506676 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 trainer/global_step‚11399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202413564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447665284717056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8257976824832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115618302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8024728416768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115608145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428190925662208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115618302 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439640644329472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86795262 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7999585751690 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447665389785600 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7999671900276 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115608145 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428190718255104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8257794321408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79529438 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439640661368832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946ìÆ šýåM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 trainer/global_step‚11399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202413564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447665284717056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8257976824832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115618302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8024728416768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115608145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428190925662208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115618302 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439640644329472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86795262 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7999585751690 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447665389785600 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7999671900276 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115608145 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428190718255104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8257794321408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79529438 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439640661368832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86795260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202403405 WCDeviceStatsMonitor.on_train_batch_end/allocated~|ùÅQ _bytes.all.allocated‚447665389785600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419932948837376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8024640387584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86795260 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433946543567040 _runtime‚4632.286277793 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202403405 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79529691 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29300094 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419932923933696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50229594 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29299844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441946232506676 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816œÎäþP2þ- _wandbruntime‚4632  _runtime‚4632s7ûÿ€ 0loss/FRt1/assignment_loss‚2.0951512509546135 1loss/SRqqt1/detection_loss‚1.1873604245483875 0loss/FBt2/detection_loss‚0.37168321986474206 %loss/total_loss‚8.940489978474226 _step‚925 2loss/SRqqt1/assignment_loss‚0.6174862664808493 /loss/FRt1/detection_loss‚0.5969042235831808 _runtime‚4632.286865154 .loss/FRt2/assignment_loss‚0.86844028519769 1loss/SRqqt2/assignment_loss‚0.615792155820385 1loss/FBt1/assignment_loss‚0.05053300251243854 epoch‚14 trainer/global_step‚11399 1loss/FBt2/assignment_loss‚0.05695123912484584 /loss/FRt2/detection_loss‚1.4670638863900756 1loss/SRqqt2/detection_loss‚0.5847045628212177 /loss/FBt1/detection_loss‚0.4284194611757994 $ _timestamp‚1.784501558376208e+09Tà €… _step‚925 2loss/SRqqt1/assignment_loss‚0.6174862664808493 /loss/FRt1/detection_loss‚0.5969042235831808 _runtime‚4632.286865154 .loss/FRt2/assignment_loss‚0.86844028519769 1loss/SRqqt2/assignment_loss‚0.615792155820385 1loss/FBt1/assignment_loss‚0.05053300251243854 epoch‚14 trainer/global_step‚11399 1loss/FBt2/assignment_loss‚0.05695123912484584 /loss/FRt2/detection_loss‚1.4670638863900756 1loss/SRqqt2/detection_loss‚0.5847045628212177 /loss/FBt1/detection_loss‚0.4284194611757994 $ _timestamp‚1.784501558376208e+09 0loss/FRt1/assignment_loss‚2.0951512509546135 1loss/SRqqt1/detection_loss‚1.1873604245483875 0loss/FBt2/detection_loss‚0.37168321986474206 %loss/total_loss‚8.940489978474226õ¹ nq‚(:æ ¹ªõÒ›´Û+ disk.md0.out‚ 141.3984375 disk.dm-1.in‚0 cpu‚26.278846595731103 disk.rbd0.out‚0' disk.nvme0c0n1.out‚68658.4990234375 disk.md0.in‚0! disk.dm-1.out‚4531.9775390625 network.recv‚ 1.34222e+06! disk.nvme0c0n1.in‚ 31.98828125% disk.nvme0n1.out‚68658.4990234375 disk.nvme1c1n1.in‚13.34375" disk.dm-0.out‚90557.4833984375# memory_percent‚98.8255262374878( proc.memory.availableMB‚ 192.42578125 proc.cpu.threads‚17) disk./.usagePercent‚76.05384661424938' disk.nvme1c1n1.out‚26732.2353515625 disk.nvme1n1.in‚13.34375 disk.dm-0.in‚ 45.33203125# proc.memory.rssMB‚ 2864.77734375) proc.memory.percent‚17.48521327972412 network.sent‚ 1.5287509e+07% disk./.usageGB‚1167.6166801452637 disk.rbd0.in‚0 disk.nvme0n1.in‚ 31.98828125% disk.nvme1n1.out‚26732.2353515625ékëš‚‚(: ¹ªõÒÖŸˆ, gpu.0.gpu‚52.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚121.627' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.54233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ6Îfƒ‚(:Û ȪõÒáÌÎ+ disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 26786.6953125 disk.md0.out‚ 141.53125) proc.memory.percent‚17.48521327972412 network.sent‚ 1.5336948e+07) disk./.usagePercent‚76.05301099614782% disk./.usageGB‚1167.6038513183594 disk.dm-1.in‚0! disk.dm-1.out‚4568.3876953125$ disk.nvme0n1.out‚68674.634765625 disk.nvme1c1n1.in‚13.34375 disk.md0.in‚0" disk.dm-0.out‚90591.3583984375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 31.98828125# memory_percent‚98.8255500793457& proc.memory.availableMB‚ 192.421875# proc.memory.rssMB‚ 2864.77734375 cpu‚26.277682725144633 proc.cpu.threads‚17 network.recv‚ 1.346134e+06 disk.dm-0.in‚ 45.33203125& disk.nvme0c0n1.out‚68674.634765625 disk.nvme0n1.in‚ 31.98828125 disk.nvme1n1.in‚13.34375" disk.nvme1n1.out‚ 26786.6953125>Š\^š„‚(: ȪõÒ»¼Œ, gpu.0.gpu‚80.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚175.835' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚58.61166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹µ+m2…- _wandbruntime‚4651  _runtime‚4651vü{džÁ _runtime‚ 4651.85276358 _step‚926 & lr-AdamW/pg1‚8.437004162524678e-05 & lr-AdamW/pg2‚8.437004162524678e-05 trainer/global_step‚11449 % _timestamp‚1.7845015779418013e+097N †Ì‡Æ & lr-AdamW/pg2‚8.437004162524678e-05 trainer/global_step‚11449 % _timestamp‚1.7845015779418013e+09 _runtime‚ 4651.85276358 _step‚926 & lr-AdamW/pg1‚8.437004162524678e-05ž;Ag2ˆ- _wandbruntime‚4651  _runtime‚4651íˆ EÝ'‰×O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87153991 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449529496343552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8057900019712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421681986638848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8057988041728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8032739275654 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚449529496343552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203259133 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87153989 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79859892 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443786117350768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8292176690176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚4651.857405597 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚441471596323840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29420352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116105142 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚429974188232704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50439804 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚449529601404928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29420088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845015779428227e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚441471613363200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8057988041728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421682011542528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8291983693824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712Y¨â'ŠÜO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚441471596323840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29420352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116105142 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚429974188232704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50439804 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚449529601404928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29420088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845015779428227e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚441471613363200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8057988041728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421682011542528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8291983693824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87153991 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449529496343552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8057900019712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421681986638848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8057988041728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8032739275654 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚449529496343552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203259133 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87153989 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79859892 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443786117350768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8292176690176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚4651.857405597Ÿ?*^o2‹- _wandbruntime‚4652  _runtime‚4652æ| ŒáM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845015782154248e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29422523 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8033501375684 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29422779 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203266207 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430010574250496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8033415227098 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚4652.126976919 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443823562182138 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203276366 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87161312 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441508962691072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203266207 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8058578129408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203276366 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87161312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79866617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79866876 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449567540820480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8292680852992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceSËl>ÞtatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8058578129408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8058666158592 trainer/global_step‚11449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116115054 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421717681796096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449567645889024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚441508979730432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441508962691072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50444097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430010362649088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116115054 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435790146955040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8292867550720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845015782154248e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29422523 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8033501375684 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29422779 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203266207 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430010574250496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8033415227098 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚4652.126976919 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443823562182138 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203276366 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87161312 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441508962691072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203266207 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8058578129408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203276366 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87161312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79866617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79866876 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449567540820480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8292680852992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8058578129408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8058666158592 trainer/global_step‚11449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116115054 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421717681796096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449567645889024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚441508979730432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441508962691072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50444097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430010362649088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116115054 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435790146955040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8292867550720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 £³õ°2Ž- _wandbruntime‚4652  _runtime‚4652A`^y ‡ /loss/FRt2/detection_loss‚1.3092444755100445 1loss/SRqqt1/detection_loss‚1.1219793035683256 _step‚929 2loss/SRqqt1/assignment_loss‚0.7144581180376312 2loss/SRqqt2/assignment_loss‚0.6072997691273653 3loss/FBt1/assignment_loss‚0.0021713221893536606 3loss/FBt2/assignment_loss‚0.0016813015599907999 /loss/FRt1/detection_loss‚0.5733746848972376 /loss/FBt2/detection_loss‚0.5169578492481167 0loss/FRt2/assignment_loss‚1.1631466416751637 epoch‚14 trainer/global_step‚11449 % _timestamp‚1.7845015782170358e+09 0loss/FRt1/assignment_loss‚2.0835543551880296 1loss/SRqqt2/detection_loss‚0.6429443017716426 /loss/FBt1/detection_loss‚0.6533978817903477 %loss/total_loss‚9.390210004563249 _runtime‚4652.131371386o:”¾Œ epoch‚14 trainer/global_step‚11449 % _timestamp‚1.7845015782170358e+09 0loss/FRt1/assignment_loss‚2.0835543551880296 1loss/SRqqt2/detection_loss‚0.6429443017716426 /loss/FBt1/detection_loss‚0.6533978817903477 %loss/total_loss‚9.390210004563249 _runtime‚4652.131371386 /loss/FRt2/detection_loss‚1.3092444755100445 1loss/SRqqt1/detection_loss‚1.1219793035683256 _step‚929 2loss/SRqqt1/assignment_loss‚0.7144581180376312 2loss/SRqqt2/assignment_loss‚0.6072997691273653 3loss/FBt1/assignment_loss‚0.0021713221893536606 3loss/FBt2/assignment_loss‚0.0016813015599907999 /loss/FRt1/detection_loss‚0.5733746848972376 /loss/FBt2/detection_loss‚0.5169578492481167 0loss/FRt2/assignment_loss‚1.1631466416751637¡Â0¤hi‘‚(:Þ תõÒôǪ+ disk.md0.in‚0 disk.md0.out‚ 142.11328125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚13.34375' disk.nvme1c1n1.out‚26859.6064453125% disk.nvme1n1.out‚26859.6064453125) proc.memory.percent‚17.48521327972412" disk.nvme0c0n1.out‚ 68716.71875 network.recv‚ 1.349518e+06" disk.dm-0.out‚90664.6220703125 disk.dm-1.out‚4608.892578125 disk.nvme0n1.in‚ 31.98828125 disk.nvme1n1.in‚13.34375$ memory_percent‚98.82571697235107 proc.cpu.threads‚17 disk.rbd0.out‚0 cpu‚26.276528498652937 network.sent‚ 1.5342959e+07) disk./.usagePercent‚76.05304826724958 disk.dm-0.in‚ 45.33203125 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 31.98828125 disk.nvme0n1.out‚ 68716.71875( proc.memory.availableMB‚ 192.39453125# proc.memory.rssMB‚ 2864.77734375% disk./.usageGB‚1167.6044235229492ßû™’‚(:Ž תõÒ䬀, gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚183.44' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.14666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£øô€2“- _wandbruntime‚4672  _runtime‚4672árq!Ȕ trainer/global_step‚11499 % _timestamp‚1.7845015981235175e+09 _runtime‚4672.034477122 _step‚930 & lr-AdamW/pg1‚8.421648779252275e-05 & lr-AdamW/pg2‚8.421648779252275e-05[•Í•Ç & lr-AdamW/pg1‚8.421648779252275e-05 & lr-AdamW/pg2‚8.421648779252275e-05 trainer/global_step‚11499 % _timestamp‚1.7845015981235175e+09 _runtime‚4672.034477122 _step‚930¢?a__2–- _wandbruntime‚4672  _runtime‚4672Å'`Ý'—×O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29544016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445663550214176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87520039 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8091837761536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431793836121600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50654307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443339931724800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8091925783552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8091837761536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚437596895314640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8066654899536 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451431752446976 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116601894 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8326886303232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80198565 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116591751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423466769404928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116591751 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204111790 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204121935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116601894 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845015981245441e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87520039 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚437596878275280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4672.035421504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431793630804480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451431752446976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8066568751062 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423466744501248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8091925783552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87520041 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443339914685440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87520041 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29544258 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8327066716672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145BÖ~Þ²˜ÜO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451431857508352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445663447026342 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50654304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451431752446976 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116601894 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8326886303232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80198565 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116591751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423466769404928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116591751 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204111790 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204121935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116601894 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845015981245441e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87520039 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚437596878275280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4672.035421504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_startY¼E^0/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431793630804480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451431752446976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8066568751062 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423466744501248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8091925783552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87520041 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443339914685440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87520041 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29544258 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8327066716672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29544016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445663550214176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87520039 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8091837761536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431793836121600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50654307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443339931724800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8091925783552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8091837761536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚437596895314640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8066654899536 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87527362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8067330851092 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50658594 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8092603900416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80204988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116611806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29546632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445700995045546 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437633750343040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204139168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚4672.306406049 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87527360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431830022715392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451469901992448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443377281052672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116601649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423502464562176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8092515871232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8067244702506 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845015983952763e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451469901992448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80205229 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423502439658496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116611806 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚443377298092032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29546394 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204139168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8092515871232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8327583056896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437633767382400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204129009 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248BªƒYì&›æM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116611806 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚443377298092032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29546394 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204139168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8092515871232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8327583056896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437633767382400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204129009 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87527362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8067330851092 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50658594 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8092603900416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80204988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116611806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29546632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445700995045546 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437633750343040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204139168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚4672.306406049 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87527360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431830022715392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451469901992448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443377281052672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116601649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423502464562176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8092515871232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8067244702506 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845015983952763e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451469901992448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80205229 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423502439658496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¤FTy;2œ- _wandbruntime‚4672  _runtime‚4672›YSÝ ƒ 2loss/SRqqt1/assignment_loss‚0.6341439244513566 0loss/FBt1/assignment_loss‚0.0347634960444518 %loss/total_loss‚9.219637346442532 trainer/global_step‚11499 % _timestamp‚1.7845015983963509e+09 _step‚933 0loss/FRt1/assignment_loss‚2.0215186729181283 /loss/FBt2/detection_loss‚0.4616251672833546 epoch‚14 _runtime‚4672.310385826 0loss/FRt2/assignment_loss‚0.8547905089884151 /loss/FRt1/detection_loss‚0.5465644334387184 .loss/FRt2/detection_loss‚1.328703563060868 /loss/FBt1/detection_loss‚1.1066625043266267 2loss/SRqqt2/assignment_loss‚0.6291376010110893 3loss/FBt2/assignment_loss‚0.0030123450770853026 1loss/SRqqt1/detection_loss‚0.9965153096818784 1loss/SRqqt2/detection_loss‚0.6021998201605595 ŸTžˆ 0loss/FRt2/assignment_loss‚0.8547905089884151 /loss/FRt1/detection_loss‚0.5465644334387184 .loss/FRt2/detection_loss‚1.328703563060868 /loss/FBt1/detection_loss‚1.1066625043266267 2loss/SRqqt2/assignment_loss‚0.6291376010110893 3loss/FBt2/assignment_loss‚0.0030123450770853026 1loss/SRqqt1/detection_loss‚0.9965153096818784 1loss/SRqqt2/detection_loss‚0.6021998201605595 2loss/SRqqt1/assignment_loss‚0.6341439244513566 0loss/FBt1/assignment_loss‚0.0347634960444518 %loss/total_loss‚9.219637346442532 trainer/global_step‚11499 % _timestamp‚1.7845015983963509e+09 _step‚933 0loss/FRt1/assignment_loss‚2.0215186729181283 /loss/FBt2/detection_loss‚0.4616251672833546 epoch‚14 _runtime‚4672.310385826¥óýQBsŸ‚(:è æªõÒ±‡Ã+ disk.dm-1.out‚4629.427734375$ memory_percent‚98.82597923278809# proc.memory.rssMB‚ 2864.77734375) proc.memory.percent‚17.48521327972412) disk./.usagePercent‚76.03757603901099( disk.nvme0c0n1.out‚68718.22314453125& disk.nvme0n1.out‚68718.22314453125" disk.dm-0.out‚90665.6357421875 proc.cpu.threads‚17( disk.nvme1c1n1.out‚26880.96044921875 disk.dm-0.in‚ 45.33203125 disk.rbd0.out‚0' proc.memory.availableMB‚ 192.3515625 cpu‚26.275376057447232 network.sent‚ 1.5404152e+07$ disk./.usageGB‚1167.366886138916! disk.nvme0c0n1.in‚ 31.98828125 disk.nvme0n1.in‚ 31.98828125 disk.nvme1n1.in‚13.34375& disk.nvme1n1.out‚26880.96044921875 disk.md0.in‚0 disk.rbd0.in‚0 network.recv‚ 1.353787e+06 disk.nvme1c1n1.in‚13.34375 disk.md0.out‚ 142.734375 disk.dm-1.in‚0h˜'c› ‚(: æªõÒ»œ, gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.781' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.92700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ãÈz2¡- _wandbruntime‚4692  _runtime‚4692à™×Ȣ _step‚934 & lr-AdamW/pg1‚8.406239998768444e-05 & lr-AdamW/pg2‚8.406239998768444e-05 trainer/global_step‚11549 % _timestamp‚1.7845016182302668e+09 _runtime‚4692.141224431_±k3ꣂ & lr-AdamW/pg1‚8.406239998768444e-05 & lr-AdamW/pg2‚8.406239998768444e-05 trainer/global_step‚11549 % _timestamp‚1.7845016182302668e+09 _runtime‚4692.141224431 _step‚934¦ zc¥2¤- _wandbruntime‚4692  _runtime‚4692e¿] ¥×O QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117088503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445208250086400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8361985355264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8361802844672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚447540879889750 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447540983077584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚453334113611776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50868807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8125775503360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433613512622592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445208233047040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_spl÷WÔÄit_bytes.large_pool.allocated‚425251527267328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚935 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87886091 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117088503 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204974592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8125863525376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445208233047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8125863525376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845016182313197e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453334008550400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50868804 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8125775503360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8100398226470 _runtime‚4692.142259643¤i»{â'¦ÜO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87886089 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204984737 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29667628 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433613305208320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845016182313197e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453334008550400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50868804 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8125775503360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8100398226470 _runtime‚4692.142259643 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117088503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445208250086400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8361985355264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8361802844672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚447540879889750 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447540983077584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚453334113611776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50868807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8125775503360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433613512622592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445208233047040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425251527267328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚935 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87886091 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117088503 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204974592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8125863525376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445208233047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8125863525376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205001970 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚11549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445245616453632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8126541642240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204991811 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117108558 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117108558 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8101074177914 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445245599414272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117098401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80543157 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80542892 % _timestamp‚1.7845016185026312e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87893410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87893412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29669798 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447578427908954 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8101160326500 _runtime‚4692.413723283 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8126453613056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447578531096900 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50873097 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8362498156544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29670060 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8362689048576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205001970 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8126453613056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439477353731040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445245616453632 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8126541642240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87893412 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425287222424576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104%uMº©æM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204991811 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117108558 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117108558 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8101074177914 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445245599414272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117098401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80543157 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80542892 % _timestamp‚1.7845016185026312e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87893410 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87893412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29669798 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447578427908954 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8101160326500 _runtime‚4692.413723283 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8126453613056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447578531096900 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50873097 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8362498156544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29670060 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8362689048576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205001970 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8126453613056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439477353731040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445245616453632 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8126541642240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87893412 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425287222424576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445245599414272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50873094 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117098401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204991811 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433649695677440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433649911473152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205001970 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚11549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445245616453632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8126541642240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¨Vù¶‘2ª- _wandbruntime‚4692  _runtime‚4692y»¼¯ «ƒ 0loss/FRt2/assignment_loss‚0.9161189136005217 2loss/SRqqt1/assignment_loss‚0.5946906002392361 2loss/SRqqt2/assignment_loss‚0.6149588562875058 1loss/FBt1/assignment_loss‚0.11755248758783171 /loss/FRt1/detection_loss‚0.5912885199404458 %loss/total_loss‚9.839264434912657 _runtime‚4692.414289924 _step‚937 .loss/FRt1/assignment_loss‚2.06807525944464 /loss/FRt2/detection_loss‚1.5079524664794124 1loss/SRqqt1/detection_loss‚1.1337663472194466 2loss/FBt2/assignment_loss‚0.015398720941851513 0loss/FBt2/detection_loss‚0.47189266232519667 1loss/SRqqt2/detection_loss‚0.6819983298468663 /loss/FBt1/detection_loss‚1.1255712709997012 epoch‚14 trainer/global_step‚11549 % _timestamp‚1.7845016185036376e+09ƒ¤‹¦¬ˆ 1loss/SRqqt1/detection_loss‚1.1337663472194466 2loss/FBt2/assignment_loss‚0.015398720941851513 0loss/FBt2/detection_loss‚0.47189266232519667 1loss/SRqqt2/detection_loss‚0.6819983298468663 /loss/FBt1/detection_loss‚1.1255712709997012 epoch‚14 trainer/global_step‚11549 % _timestamp‚1.7845016185036376e+09 0loss/FRt2/assignment_loss‚0.9161189136005217 2loss/SRqqt1/assignment_loss‚0.5946906002392361 2loss/SRqqt2/assignment_loss‚0.6149588562875058 1loss/FBt1/assignment_loss‚0.11755248758783171 /loss/FRt1/detection_loss‚0.5912885199404458 %loss/total_loss‚9.839264434912657 _runtime‚4692.414289924 _step‚937 .loss/FRt1/assignment_loss‚2.06807525944464 /loss/FRt2/detection_loss‚1.5079524664794124©B,²¾v­‚(:ë õªõÒ±ã¶+ disk.nvme1c1n1.in‚13.34375 cpu‚26.274234318649146 disk.md0.in‚0# disk.dm-0.out‚90683.50244140625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 31.98828125& disk.nvme0n1.out‚68734.01806640625$ memory_percent‚98.82755279541016% proc.memory.availableMB‚ 192.09375) proc.memory.percent‚17.48521327972412 network.sent‚ 1.5467804e+07 network.recv‚ 1.358314e+06 disk.md0.out‚ 143.0859375" disk.dm-1.out‚4649.52490234375( disk.nvme0c0n1.out‚68734.01806640625( disk.nvme1c1n1.out‚26903.87646484375 disk.nvme1n1.in‚13.34375& disk.nvme1n1.out‚26903.87646484375 disk.dm-0.in‚ 45.33203125# proc.memory.rssMB‚ 2864.77734375 proc.cpu.threads‚17) disk./.usagePercent‚76.03798403333829% disk./.usageGB‚1167.3731498718262 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 31.98828125úE'Eš®‚(: õªõÒ´ÈŠ, gpu.0.gpu‚29.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.561' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.85366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$Î7Œw¯‚(:ì „«õÒÐñÃ,# proc.memory.rssMB‚ 2864.77734375( disk.nvme1c1n1.out‚26908.94775390625 disk.dm-0.in‚ 45.33203125 disk.dm-1.in‚0" disk.dm-1.out‚4653.97021484375! disk.nvme0c0n1.in‚ 31.98828125$ memory_percent‚99.22077655792236( proc.memory.availableMB‚ 127.66796875 proc.cpu.threads‚17 network.sent‚ 1.5530835e+07 disk.nvme0n1.in‚ 31.98828125& disk.nvme0n1.out‚68741.88037109375 disk.nvme1c1n1.in‚13.34375 disk.nvme1n1.in‚13.34375 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.362076e+06) disk./.usagePercent‚76.03851651314551% disk./.usageGB‚1167.3813247680664& disk.nvme1n1.out‚26908.94775390625# disk.dm-0.out‚90690.91064453125( disk.nvme0c0n1.out‚68741.88037109375) proc.memory.percent‚17.48521327972412 cpu‚26.273244711096066 disk.md0.out‚ 143.59375×H§›°‚(: „«õÒï«ð, gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚112.421' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.47366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ȳ¶2±- _wandbruntime‚4712  _runtime‚4712VRÚ)Ų¿ % lr-AdamW/pg1‚8.39077806787608e-05 % lr-AdamW/pg2‚8.39077806787608e-05 trainer/global_step‚11599 $ _timestamp‚1.784501638339843e+09 _runtime‚4712.250841703 _step‚938(­ëÊ³Ä $ _timestamp‚1.784501638339843e+09 _runtime‚4712.250841703 _step‚938 % lr-AdamW/pg1‚8.39077806787608e-05 % lr-AdamW/pg2‚8.39077806787608e-05 trainer/global_step‚11599ªT+Æ2´- _wandbruntime‚4712  _runtime‚4712úà¥YÝ'µ×O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4712.255461403 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚455236264653824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8159713245184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8396929590784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117595398 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80875041 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205837394 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88252139 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447076568448000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449418415940992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29791734 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚11599 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117595398 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29791474 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8159713245184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88252141 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117585255 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚455236369715200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447076551408640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435433003111936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8396742885888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚455236264653824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8159713245184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8396929590784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117595398 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80875041 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205837394 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88252139 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447076568448000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449418415940992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29791734 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚11599 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117595398 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29791474 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8159713245184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88252141 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117585255 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚455236369715200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447076551408640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435433003111936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8396742885888 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80881532 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449455860772362 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 81349036533rNÐÂs22 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8397449560064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447113934815232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205864772 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117605310 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51087594 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441320974158400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚455274309130752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80881768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚455274414199296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117595153 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8134989801908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8160391354880 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚455274414199296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449455963960308 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784501638612575e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205864772 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117605310 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51087594 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441320974158400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚455274309130752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80881768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚455274414199296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117595153 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8134989801908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8160391354880 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚455274414199296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449455963960308 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784501638612575e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80881532 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449455860772362 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8134903653322¬¢¨Èò2º- _wandbruntime‚4712  _runtime‚4712ŠŸ? »ƒ 2loss/SRqqt1/assignment_loss‚0.6141173454843437 % _timestamp‚1.7845016386136627e+09 _step‚941 2loss/SRqqt2/assignment_loss‚0.5881669925917683 1loss/FBt1/assignment_loss‚0.01071547036699485 3loss/FBt2/assignment_loss‚7.545321646585488e-05 /loss/FBt1/detection_loss‚0.8043657578527927 0loss/FBt2/detection_loss‚0.36264881444410146 #loss/total_loss‚9.0941643851007 /loss/FRt1/detection_loss‚0.5592956798405645 /loss/FRt2/detection_loss‚1.5119589052961342 1loss/SRqqt1/detection_loss‚0.9804385957240269 1loss/SRqqt2/detection_loss‚0.6255408413039156 epoch‚14 /loss/FRt1/assignment_loss‚2.138592286499179 0loss/FRt2/assignment_loss‚0.8982482424804142 trainer/global_step‚11599 _runtime‚4712.524413678Pùмˆ 2loss/SRqqt2/assignment_loss‚0.5881669925917683 1loss/FBt1/assignment_loss‚0.01071547036699485 3loss/FBt2/assignment_loss‚7.545321646585488e-05 /loss/FBt1/detection_loss‚0.8043657578527927 0loss/FBt2/detection_loss‚0.36264881444410146 #loss/total_loss‚9.0941643851007 /loss/FRt1/detection_loss‚0.5592956798405645 /loss/FRt2/detection_loss‚1.5119589052961342 1loss/SRqqt1/detection_loss‚0.9804385957240269 1loss/SRqqt2/detection_loss‚0.6255408413039156 epoch‚14 /loss/FRt1/assignment_loss‚2.138592286499179 0loss/FRt2/assignment_loss‚0.8982482424804142 trainer/global_step‚11599 _runtime‚4712.524413678 2loss/SRqqt1/assignment_loss‚0.6141173454843437 % _timestamp‚1.7845016386136627e+09 _step‚941­Éæyu½‚(:ê “«õÒóÚÌ+ network.recv‚ 1.365398e+06" disk.dm-1.out‚4657.23486328125' disk.nvme0c0n1.out‚68815.5517578125 disk.nvme1c1n1.in‚13.34375( disk.nvme1c1n1.out‚26914.31494140625 cpu‚26.272126627488507% disk./.usageGB‚1167.6990585327148 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 31.98828125 disk.nvme1n1.in‚13.34375 disk.md0.in‚0 disk.md0.out‚ 144.046875" disk.dm-0.out‚90765.7060546875) disk./.usagePercent‚76.05921241053345 disk.rbd0.in‚0% disk.nvme0n1.out‚68815.5517578125 disk.dm-0.in‚ 45.33203125( proc.memory.availableMB‚ 127.89453125) proc.memory.percent‚17.48521327972412 proc.cpu.threads‚17 network.sent‚ 1.5533757e+07 disk.nvme0n1.in‚ 31.98828125& disk.nvme1n1.out‚26914.31494140625$ memory_percent‚99.21939373016357# proc.memory.rssMB‚ 2864.77734375†T›6š¾‚(: “«õÒèÈý+ gpu.0.gpu‚47.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.444' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.48133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁR‘û2¿- _wandbruntime‚4732  _runtime‚4732,iÇaÇÀÁ & lr-AdamW/pg1‚8.375263234229389e-05 & lr-AdamW/pg2‚8.375263234229389e-05 trainer/global_step‚11649 $ _timestamp‚1.784501658187153e+09 _runtime‚4732.099400631 _step‚942 ü)ÌÁÆ $ _timestamp‚1.784501658187153e+09 _runtime‚4732.099400631 _step‚942 & lr-AdamW/pg1‚8.375263234229389e-05 & lr-AdamW/pg2‚8.375263234229389e-05 trainer/global_step‚11649®²¿ø2Â- _wandbruntime‚4732  _runtime‚4732Ö!µÝ'Ã×O JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8168143325760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118092150 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457138520757248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428821018088448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845016581879234e+09 _step‚943 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚443127688439280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8168057177286 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451295848804400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51297807 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443127705478640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457138625818624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8193739009024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8193650987008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451295745616566 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88618189 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81213399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428821042992128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206700196 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268>^Ð}ÄÜO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚443127688439280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8168057177286 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451295848804400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51297807 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443127705478640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457138625818624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8193739009024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8193650987008 HBDe¼ (ÇviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451295745616566 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88618189 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81213399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428821042992128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206700196 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8168143325760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118092150 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457138520757248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428821018088448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845016581879234e+09 _step‚943 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143¯I~jâ2Å- _wandbruntime‚4732  _runtime‚4732T9¬³ç&ÆáM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81220105 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8194329096704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚4732.369629976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81220379 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437289167751168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8432454505472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118102062 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448982253176832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206717415 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437289373061120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29918011 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428856738149376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88625510 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118091905 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451333396823716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457176670302720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8168819277316 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457176565234176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8168733128730 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443164560507040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚11649 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457176670302720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118091905 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448982253176832 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206717415 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206727574 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448982236137472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8194417125888 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚11649 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457176670302720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118091905 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448982253176832 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206717415 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206727574 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448982236137472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8194417125888 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81220105 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8194329096704 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚4732.369629976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81220379 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437289167751168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8432454505472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118102062 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448982253176832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206717415 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437289373061120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29918011 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428856738149376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88625510 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118091905 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451333396823716 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457176670302720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8168819277316 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330°ËŠ&œ2È- _wandbruntime‚4732  _runtime‚4732~ÚF5 Ƀ % _timestamp‚1.7845016584595766e+09 2loss/SRqqt2/assignment_loss‚0.5371784434454694 /loss/FRt2/detection_loss‚1.4705007473289118 /loss/FBt2/detection_loss‚0.4350668855278367 epoch‚14 $loss/total_loss‚9.22020229081322 0loss/FRt2/assignment_loss‚0.9600892996695971 1loss/SRqqt1/assignment_loss‚0.745252005639486 trainer/global_step‚11649 _step‚945 _runtime‚4732.370243677 /loss/FBt1/detection_loss‚0.5445281356805936 0loss/FRt1/assignment_loss‚2.1258046318832626 3loss/FBt1/assignment_loss‚0.0005139588132992271 2loss/FBt2/assignment_loss‚0.004014633807700003 /loss/FRt1/detection_loss‚0.5555032101809545 1loss/SRqqt1/detection_loss‚1.1701520215421437 0loss/SRqqt2/detection_loss‚0.671598317293965rîjʈ 0loss/FRt1/assignment_loss‚2.1258046318832626 3loss/FBt1/assignment_loss‚0.0005139588132992271 2loss/FBt2/assignment_loss‚0.004014633807700003 /loss/FRt1/detection_loss‚0.5555032101809545 1loss/SRqqt1/detection_loss‚1.1701520215421437 0loss/SRqqt2/detection_loss‚0.671598317293965 % _timestamp‚1.7845016584595766e+09 2loss/SRqqt2/assignment_loss‚0.5371784434454694 /loss/FRt2/detection_loss‚1.4705007473289118 /loss/FBt2/detection_loss‚0.4350668855278367 epoch‚14 $loss/total_loss‚9.22020229081322 0loss/FRt2/assignment_loss‚0.9600892996695971 1loss/SRqqt1/assignment_loss‚0.745252005639486 trainer/global_step‚11649 _step‚945 _runtime‚4732.370243677 /loss/FBt1/detection_loss‚0.5445281356805936±VxmxË‚(:í ¢«õÒÝÄô+ proc.cpu.threads‚17 network.sent‚ 1.5609972e+07 network.recv‚ 1.369265e+06 disk.md0.out‚ 144.4296875# disk.dm-0.out‚90909.99267578125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚68930.11669921875& disk.nvme0n1.out‚68930.11669921875' proc.memory.availableMB‚ 128.0703125) disk./.usagePercent‚76.03954569250224% disk./.usageGB‚1167.3971252441406 disk.md0.in‚0 disk.dm-0.in‚ 45.33203125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 31.98828125( disk.nvme1c1n1.out‚26944.93310546875 disk.nvme1n1.in‚13.34375" disk.dm-1.out‚4657.30126953125! disk.nvme0c0n1.in‚ 31.98828125 disk.nvme1c1n1.in‚13.34375& disk.nvme1n1.out‚26944.93310546875) proc.memory.percent‚17.48521327972412 cpu‚26.271053838808612 disk.rbd0.in‚0$ memory_percent‚99.21832084655762# proc.memory.rssMB‚ 2864.77734375Ç¥cãšÌ‚(: ¢«õÒ¢ð£, gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚186.13' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.04333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g­JÕ2Í- _wandbruntime‚4751  _runtime‚4751¿‚>¾ÈΠ_step‚946 & lr-AdamW/pg1‚8.359695746329928e-05 & lr-AdamW/pg2‚8.359695746329928e-05 trainer/global_step‚11699 % _timestamp‚1.7845016779476728e+09 _runtime‚4751.858764662Ý™ó7ÍÏÇ _step‚946 & lr-AdamW/pg1‚8.359695746329928e-05 & lr-AdamW/pg2‚8.359695746329928e-05 trainer/global_step‚11699 % _timestamp‚1.7845016779476728e+09 _runtime‚4751.858764662²˜…¹32Ð- _wandbruntime‚4751  _runtime‚4751î˜Á~Ñ×O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88984241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚459040776860672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚459040776860672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8201972801168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_Ÿ6 N_#batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453173178479974 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8227676750848 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30040944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444971308866640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453173281667808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207562998 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8466939423744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8227588728832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450813205171200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8201886652694 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51512307 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚4751.860747615 _step‚947 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118588902 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118588902 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81552978 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8466754816000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚249036808nÐâ'ÒÜO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8466754816000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88984241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚459040776860672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚459040776860672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8201972801168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453173178479974 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8227676750848 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30040944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444971308866640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453173281667808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207562998 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8466939423744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8227588728832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450813205171200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8201886652694 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51512307 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚4751.860747615 _step‚947 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118588902 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118588902 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81552978³œî€O2Ó- _wandbruntime‚4752  _runtime‚4752¡DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81559806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8202648752724 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8467455493632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8228266838528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453210726499178 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8202562604138 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51516594 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88991560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8228354867712 _runtime‚4752.117289979 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207590376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118598814 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430641471108096 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450850554499072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30043212 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450850571538432 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚450850571538432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118598814 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88991562 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439108926601728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784501678206114e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445008180934400 trainer/global_step‚11699 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207590376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8228266838528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51516597 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459078821337600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207580217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118588657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30043470 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8467637997056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459078926406144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81560067 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430641496011776 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚450850571538432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118598814 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88991562 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439108926601728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784501678206114e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445008180934400 trainer/global_step‚11699 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207590376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 82®.g28266838528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51516597 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459078821337600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207580217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118588657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30043470 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8467637997056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459078926406144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81560067 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430641496011776 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81559806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8202648752724 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8467455493632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8228266838528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453210726499178 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8202562604138 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51516594 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88991560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8228354867712 _runtime‚4752.117289979 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207590376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118598814 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430641471108096 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450850554499072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30043212 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258´w+2Ö- _wandbruntime‚4752  _runtime‚4752u3K¢ ׄ epoch‚14 2loss/SRqqt1/assignment_loss‚0.5865140835647797 2loss/SRqqt2/assignment_loss‚0.6242472315896558 /loss/FRt2/detection_loss‚1.6012565137869026 .loss/FBt1/detection_loss‚1.214491124264896 /loss/FRt1/detection_loss‚0.5899916542047613 trainer/global_step‚11699 &loss/total_loss‚10.012197218618644 0loss/FRt1/assignment_loss‚2.0144452267480126 2loss/FBt1/assignment_loss‚0.014553781317469353 0loss/SRqqt1/detection_loss‚1.060307593523839 /loss/FBt2/detection_loss‚0.5211340426589901 % _timestamp‚1.7845016782075522e+09 0loss/FRt2/assignment_loss‚1.0506255637466637 2loss/FBt2/assignment_loss‚0.010341329631369035 1loss/SRqqt2/detection_loss‚0.7242890735813055 _runtime‚4752.118210921 _step‚949źȵ؉ 0loss/FRt2/assignment_loss‚1.0506255637466637 2loss/FBt2/assignment_loss‚0.010341329631369035 1loss/SRqqt2/detection_loss‚0.7242890735813055 _runtime‚4752.118210921 _step‚949 epoch‚14 2loss/SRqqt1/assignment_loss‚0.5865140835647797 2loss/SRqqt2/assignment_loss‚0.6242472315896558 /loss/FRt2/detection_loss‚1.6012565137869026 .loss/FBt1/detection_loss‚1.214491124264896 /loss/FRt1/detection_loss‚0.5899916542047613 trainer/global_step‚11699 &loss/total_loss‚10.012197218618644 0loss/FRt1/assignment_loss‚2.0144452267480126 2loss/FBt1/assignment_loss‚0.014553781317469353 0loss/SRqqt1/detection_loss‚1.060307593523839 /loss/FBt2/detection_loss‚0.5211340426589901 % _timestamp‚1.7845016782075522e+09µ…xu+iÙ‚(:Þ ±«õÒ”Íø+ disk.dm-1.in‚0 disk.dm-1.out‚4657.806640625 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 68981.765625$ memory_percent‚99.22008514404297% proc.memory.availableMB‚ 127.78125 cpu‚26.27208832204472 disk.dm-0.in‚ 45.33203125 disk.nvme0n1.in‚ 31.98828125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2864.77734375 proc.cpu.threads‚17! disk.nvme0n1.out‚ 68981.765625& disk.nvme1n1.out‚26976.39111328125 disk.md0.out‚ 144.8046875! disk.nvme0c0n1.in‚ 31.98828125) proc.memory.percent‚17.48521327972412 disk.md0.in‚0# disk.dm-0.out‚90991.77490234375 network.sent‚ 1.5671163e+07 network.recv‚ 1.373397e+06) disk./.usagePercent‚76.04158640956086% disk./.usageGB‚1167.4284553527832 disk.nvme1c1n1.in‚13.34375( disk.nvme1c1n1.out‚26976.39111328125 disk.nvme1n1.in‚13.34375Ϧ‚ŽÚ‚(:ƒ ±«õÒЀ‹, gpu.0.gpu‚86.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚115.74' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.58 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é>(qÛ‚(:æ À«õÒ –Ô+ cpu‚26.273381275932156) disk./.usagePercent‚76.04159063361907% disk./.usageGB‚1167.4285202026367 disk.dm-1.out‚4657.908203125! disk.nvme0c0n1.in‚ 31.98828125' disk.nvme0c0n1.out‚68983.1357421875 disk.nvme1c1n1.in‚13.34375$ memory_percent‚99.21879768371582) proc.memory.percent‚17.48521327972412 disk.dm-0.in‚ 45.33203125 disk.nvme0n1.in‚ 31.98828125 disk.md0.in‚0' proc.memory.availableMB‚ 127.9921875# proc.memory.rssMB‚ 2864.77734375 proc.cpu.threads‚17% disk.nvme0n1.out‚68983.1357421875 network.recv‚ 1.377042e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚13.34375% disk.nvme1n1.out‚26976.8232421875 disk.md0.out‚ 145.04296875 network.sent‚ 1.5720547e+07! disk.dm-0.out‚90992.947265625 disk.rbd0.in‚0' disk.nvme1c1n1.out‚26976.8232421875w7p šÜ‚(: À«õÒ‰â—, gpu.0.gpu‚58.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.348' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.78266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å›2Ý- _wandbruntime‚4771  _runtime‚4771÷Ù;ÈÞ & lr-AdamW/pg1‚8.344075853522609e-05 & lr-AdamW/pg2‚8.344075853522609e-05 trainer/global_step‚11749 % _timestamp‚1.7845016974107454e+09 _runtime‚4771.321827929 _step‚950ðêfAÍßÇ & lr-AdamW/pg2‚8.344075853522609e-05 trainer/global_step‚11749 % _timestamp‚1.7845016974107454e+09 _runtime‚4771.321827929 _step‚950 & lr-AdamW/pg1‚8.344075853522609e-05¶¡„ú—2à- _wandbruntime‚4771  _runtime‚4771"×´Ü'áÖO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚452681506493440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚452681523532800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208435945 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455050611343382 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚4771.323799183 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30164689 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8261614492672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8501932150272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚446814895215280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚452681523532800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81891493 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119085654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440892490867200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460943138025472 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460943032964096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460943032964096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚432390533813248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _step‚951 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460943138025472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119075511 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89350289 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119075511 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8261526470656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8261526470656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119085654 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51726807 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8235802276576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89350289 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚11749k"Æá'âÛO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460943032964096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460943032964096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚432390533813248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _step‚951 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460943138025472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119075511 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89350289 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119075511 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8261526470656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8261526470656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119085654 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51726807 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8235802276576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89350289 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚11749 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚452681506493440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚452681523532800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208435945 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455050611343382 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚4771.323799183 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30164689 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8261614492672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8501932150272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚446814895215280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚452681523532800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81891493 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119085654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440892490867200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460943138025472 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8262292609536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81898228 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208453178 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119095566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119095566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51731097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81898477 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432426228970496 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚452718872860672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119085409 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚432426253874176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446851767283040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8262204580352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440928889771008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚11749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89357612 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8262292609536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460981182509568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455088159362586 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8236478228132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30167134 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440928676072448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8502635896832 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208443019 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446851784322400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845016976667924e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119085409‡¹Zì&åæM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119085409 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚432426253874176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446851767283040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8262204580352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440928889771008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚11749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89357612 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8262292609536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460981182509568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455088159362586 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8236478228132 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30167134 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440928676072448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8502635896832 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208443019 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446851784322400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845016976667924e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119085409 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8262292609536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81898228 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208453178 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119095566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119095566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51731097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81898477 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432426228970496 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚452718872860672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246¸IjÈ2æ- _wandbruntime‚4771  _runtime‚4771¹Ã¸–ç‚ 0loss/FRt2/assignment_loss‚0.8625130772124976 /loss/FRt1/detection_loss‚0.5027689412301641 1loss/SRqqt2/detection_loss‚0.6257745999211878 /loss/FBt1/detection_loss‚0.4723816876985917 /loss/FBt2/detection_loss‚0.3920076236709196 % _timestamp‚1.7845016976682858e+09 0loss/FRt1/assignment_loss‚1.8935855081839155 epoch‚14 trainer/global_step‚11749 _step‚953 2loss/SRqqt1/assignment_loss‚0.7434827726804683 2loss/SRqqt2/assignment_loss‚0.5922649714130692 0loss/FBt1/assignment_loss‚0.0914822742118926 3loss/FBt2/assignment_loss‚0.0009683716433934119 /loss/FRt2/detection_loss‚1.3797725555687066 1loss/SRqqt1/detection_loss‚0.8334515234017932 #loss/total_loss‚8.3904539068366 _runtime‚4771.579544936wê% è‡ trainer/global_step‚11749 _step‚953 2loss/SRqqt1/assignment_loss‚0.7434827726804683 2loss/SRqqt2/assignment_loss‚0.5922649714130692 0loss/FBt1/assignment_loss‚0.0914822742118926 3loss/FBt2/assignment_loss‚0.0009683716433934119 /loss/FRt2/detection_loss‚1.3797725555687066 1loss/SRqqt1/detection_loss‚0.8334515234017932 #loss/total_loss‚8.3904539068366 _runtime‚4771.579544936 0loss/FRt2/assignment_loss‚0.8625130772124976 /loss/FRt1/detection_loss‚0.5027689412301641 1loss/SRqqt2/detection_loss‚0.6257745999211878 /loss/FBt1/detection_loss‚0.4723816876985917 /loss/FBt2/detection_loss‚0.3920076236709196 % _timestamp‚1.7845016976682858e+09 0loss/FRt1/assignment_loss‚1.8935855081839155 epoch‚14¹ÔjZdé‚(:Ù Ï«õÒɈ, proc.cpu.threads‚17( disk.nvme1c1n1.out‚26981.61572265625% disk./.usageGB‚1167.4284133911133 disk.dm-1.out‚ 4661.45703125 disk.nvme0n1.in‚ 31.98828125 disk.nvme0n1.out‚ 68987.65625 disk.md0.out‚ 145.71875 cpu‚26.273720305231112" disk.nvme0c0n1.out‚ 68987.65625 disk.md0.in‚0$ memory_percent‚99.21455383300781# proc.memory.rssMB‚ 2864.77734375) proc.memory.percent‚17.48521327972412 disk.nvme1c1n1.in‚13.34375 network.sent‚ 1.5739405e+07 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 31.98828125 disk.dm-0.in‚ 45.33203125$ proc.memory.availableMB‚128.6875 network.recv‚ 1.380535e+06) disk./.usagePercent‚76.04158367634673 disk.rbd0.in‚0 disk.nvme1n1.in‚13.34375& disk.nvme1n1.out‚26981.61572265625# disk.dm-0.out‚90997.28662109375Ë—W›ê‚(: Ï«õÒŒ¬², gpu.0.gpu‚77.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚199.664' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.55466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%´¨#2ë- _wandbruntime‚4790  _runtime‚4790O!Åì¿ % lr-AdamW/pg1‚8.32840380599172e-05 % lr-AdamW/pg2‚8.32840380599172e-05 trainer/global_step‚11799 $ _timestamp‚1.784501716476635e+09 _runtime‚4790.387592037 _step‚9547àÊíÄ % lr-AdamW/pg1‚8.32840380599172e-05 % lr-AdamW/pg2‚8.32840380599172e-05 trainer/global_step‚11799 $ _timestamp‚1.784501716476635e+09 _runtime‚4790.387592037 _step‚954º¹-ü2î- _wandbruntime‚4790  _runtime‚4790bpÜgÝ'ï×O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462845289067520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚448658515642640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119582406 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434175316579328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚454549824855040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚454549841894400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89716339 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8295552234496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119572263 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚442712011660800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434175291675648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8295464212480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462845289067520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448658498603280 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209288602 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442712210686464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 trainer/global_step‚11799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119582406 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454549841894400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456928147394624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845017164775093e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119572263 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89716341 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209298747 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30289319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30289070 _step‚955 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8536719985152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8536894107136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51941304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462845394128896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚454549824855040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82230626 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8295552234496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8269545603510 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8269631751984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4790.3913989743V¬M°ðÜO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDevic¥ø>2'eStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8295464212480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462845289067520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448658498603280 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209288602 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442712210686464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 trainer/global_step‚11799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119582406 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454549841894400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456928147394624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845017164775093e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119572263 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89716341 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209298747 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30289319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30289070 _step‚955 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8536719985152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8536894107136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51941304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462845394128896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚454549824855040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82230626 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8295552234496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8269545603510 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8269631751984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4790.391398974 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462845289067520 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚448658515642640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119582406 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434175316579328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚454549824855040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚454549841894400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89716339 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8295552234496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119572263 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚442712011660800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434175291675648 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0»!]1ß2ñ- _wandbruntime‚4790  _runtime‚47905½J|å&òßM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845017167380219e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚454587191222272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119592318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30291570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456965695413940 _runtime‚ 4790.6491033 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119582161 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8537592406016 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434211011736576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448695370671040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8270307703540 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119582161 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82237425 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442748604142592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51945597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30291828 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚11799 _step‚956 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚454587191222272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8296230351360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454587208261632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209315980 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8270221554954 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8296142322176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8537420388352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89723660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51945594 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89723662 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119592318 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82237164 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89723660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434210986832896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448695387710400´S?xê&óäM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8537592406016 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434211011736576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448695370671040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8270307703540 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119582161 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82237425 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442748604142592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51945597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30291828 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚11799 _step‚956 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚454587191222272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8296230351360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454587208261632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209315980 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8270221554954 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8296142322176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8537420388352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89723660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51945594 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89723662 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119592318 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82237164 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89723660 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434210986832896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448695387710400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845017167380219e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚454587191222272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119592318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30291570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456965695413940 _runtime‚ 4790.6491033 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119582161 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0¼½Äš2ô- _wandbruntime‚4790  _runtime‚4790¯¾ö"õ‚ &loss/total_loss‚10.121504444458825 _runtime‚4790.649563701 _step‚957 0loss/FRt1/assignment_loss‚2.0690592337738383 2loss/SRqqt1/assignment_loss‚0.5985087501408154 2loss/SRqqt2/assignment_loss‚0.6005022754939063 1loss/SRqqt1/detection_loss‚1.1784108037709433 0loss/FRt2/assignment_loss‚1.1806927694214715 1loss/FBt2/assignment_loss‚0.01766591854004967 epoch‚14 $ _timestamp‚1.784501716738942e+09 2loss/FBt1/assignment_loss‚0.003669112806689901 .loss/FRt1/detection_loss‚0.573963096988116 /loss/FRt2/detection_loss‚1.5245689539466467 1loss/SRqqt2/detection_loss‚0.8073475668778431 .loss/FBt1/detection_loss‚0.791605300846554 /loss/FBt2/detection_loss‚0.7755106618519519 trainer/global_step‚117999‰“ ö‡ _step‚957 0loss/FRt1/assignment_loss‚2.0690592337738383 2loss/SRqqt1/assignment_loss‚0.5985087501408154 2loss/SRqqt2/assignment_loss‚0.6005022754939063 1loss/SRqqt1/detection_loss‚1.1784108037709433 0loss/FRt2/assignment_loss‚1.1806927694214715 1loss/FBt2/assignment_loss‚0.01766591854004967 epoch‚14 $ _timestamp‚1.784501716738942e+09 2loss/FBt1/assignment_loss‚0.003669112806689901 .loss/FRt1/detection_loss‚0.573963096988116 /loss/FRt2/detection_loss‚1.5245689539466467 1loss/SRqqt2/detection_loss‚0.8073475668778431 .loss/FBt1/detection_loss‚0.791605300846554 /loss/FBt2/detection_loss‚0.7755106618519519 trainer/global_step‚11799 &loss/total_loss‚10.121504444458825 _runtime‚4790.649563701½ÍŠÔe÷‚(:Ú Þ«õÒÍ»Ñ+ disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 31.98828125$ disk.nvme0n1.out‚68997.341796875 disk.md0.out‚ 146.109375 disk.dm-0.out‚ 91006.7421875' proc.memory.availableMB‚ 128.6796875# proc.memory.rssMB‚ 2864.77734375& disk.nvme1c1n1.out‚26982.373046875 disk.nvme1n1.in‚13.34375 disk.dm-0.in‚ 45.33203125$ memory_percent‚99.21460151672363) disk./.usagePercent‚76.04161423865018& disk.nvme0c0n1.out‚68997.341796875$ disk.nvme1n1.out‚26982.373046875) proc.memory.percent‚17.48521327972412 cpu‚26.272768307650168 proc.cpu.threads‚17 network.sent‚ 1.578772e+07 network.recv‚ 1.384687e+06$ disk./.usageGB‚1167.428882598877 disk.dm-1.out‚ 4661.5859375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 31.98828125 disk.nvme1c1n1.in‚13.34375 disk.md0.in‚0­Ìâø‚(: Þ«õÒ¶»¯, gpu.0.gpu‚70.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.602' gpu.0.enforcedPowerv@>=¸LimitWatts‚300.0( gpu.0.powerPercent‚37.20066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ šC2ù- _wandbruntime‚4809  _runtime‚4809(I;ÇúÁ & lr-AdamW/pg1‚8.312679854756912e-05 & lr-AdamW/pg2‚8.312679854756912e-05 trainer/global_step‚11849 $ _timestamp‚1.784501735138077e+09 _runtime‚4809.049020605 _step‚958½àÌûÆ _runtime‚4809.049020605 _step‚958 & lr-AdamW/pg1‚8.312679854756912e-05 & lr-AdamW/pg2‚8.312679854756912e-05 trainer/global_step‚11849 $ _timestamp‚1.784501735138077e+09¾/’1œ2ü- _wandbruntime‚4809  _runtime‚4809ö†¼ÿÝ'ý×O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456418160256000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464747545170944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52155807 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464747650232320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456418160256000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30413729 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8329401954304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8303375078918 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8329401954304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90082391 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8329489976320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82569280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90082391 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8571674504192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82569536 % _timestamp‚1.7845017351389666e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90082389 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90082389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30413476 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464747650232320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435960074441728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435960049538048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8571850723328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210151404 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464747545170944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚450502119030640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 83034612273920WgYâ'þÜO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456418160256000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464747545170944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52155807 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464747650232320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456418160256000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30413729 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8329401954304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8303375078918 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8329401954304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90082391 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8329489976320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82569280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90082391 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8571674504192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82569536 % _timestamp‚1.7845017351389666e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90082389 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90082389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30413476 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464747650232320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435960074441728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435960049538048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8571850723328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210151404 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464747545170944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚450502119030640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8303461227392¿[å¨Ö2ÿ- _wandbruntime‚4809  _runtime‚4809{2iç&€ áM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8330080064000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450538991098400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82576059 _step‚960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435995769598976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90089712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚458843128277348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30416231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450538974059040 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90089710 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8330080064000 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚11849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8572365875712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120078913 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8304137178948 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210168623 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120089070 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456455526623232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82576328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52160097 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456455509583872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8330168093184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90089712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210178782 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444568110571008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚4809.619595252 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464785694716416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8572560962048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845017357084575e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210168623 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 69373788160oL%3 æM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8330168093184 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90089712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210178782 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444568110571008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚4809.619595252 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464785694716416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8572560962048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/resHÕ•¹ erved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845017357084575e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210168623 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8330080064000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450538991098400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82576059 _step‚960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435995769598976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90089712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚458843128277348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30416231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450538974059040 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90089710 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8330080064000 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚11849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8572365875712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120078913 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8304137178948 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210168623 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120089070 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456455526623232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82576328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52160097 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456455509583872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928À[­×å2‚ - _wandbruntime‚4809  _runtime‚4809A~wƒ  /loss/FBt1/detection_loss‚0.7894873566586863 0loss/FRt1/assignment_loss‚1.9999992278924428 2loss/SRqqt1/assignment_loss‚0.5933146864462358 1loss/FBt1/assignment_loss‚0.00839887008275591 epoch‚14 % _timestamp‚1.7845017357094724e+09 _runtime‚4809.620164163 _step‚961 1loss/SRqqt2/assignment_loss‚0.511843247082632 0loss/FBt2/assignment_loss‚0.1020430303035302 /loss/FRt1/detection_loss‚0.5449906967353099 1loss/SRqqt2/detection_loss‚0.6381255895146271 /loss/FBt2/detection_loss‚0.4160162511963289 trainer/global_step‚11849 1loss/SRqqt1/detection_loss‚1.0819676350515621 %loss/total_loss‚8.907456065991969 0loss/FRt2/assignment_loss‚0.8810244599978129 /loss/FRt2/detection_loss‚1.3402450150300447ý/^ª „ † 0loss/FRt2/assignment_loss‚0.8810244599978129 /loss/FRt2/detection_loss‚1.3402450150300447 /loss/FBt1/detection_loss‚0.7894873566586863 0loss/FRt1/assignment_loss‚1.9999992278924428 2loss/SRqqt1/assignment_loss‚0.5933146864462358 1loss/FBt1/assignment_loss‚0.00839887008275591 epoch‚14 % _timestamp‚1.7845017357094724e+09 _runtime‚4809.620164163 _step‚961 1loss/SRqqt2/assignment_loss‚0.511843247082632 0loss/FBt2/assignment_loss‚0.1020430303035302 /loss/FRt1/detection_loss‚0.5449906967353099 1loss/SRqqt2/detection_loss‚0.6381255895146271 /loss/FBt2/detection_loss‚0.4160162511963289 trainer/global_step‚11849 1loss/SRqqt1/detection_loss‚1.0819676350515621 %loss/total_loss‚8.907456065991969Á‚fp… ‚(:å í«õÒ¹œË+ disk.dm-0.in‚ 45.33203125# disk.dm-0.out‚91252.99853515625 cpu‚26.271869637194904 disk.nvme1n1.in‚13.34375! disk.nvme0c0n1.in‚ 31.98828125$ memory_percent‚99.21512603759766) proc.memory.percent‚17.48521327972412 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 4661.828125% disk./.usageGB‚1167.4291610717773' disk.nvme1c1n1.out‚27081.2958984375% disk.nvme1n1.out‚27081.2958984375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚69145.85693359375 disk.nvme0n1.in‚ 31.98828125 network.sent‚ 1.5863141e+07) disk./.usagePercent‚76.04163237725304 disk.md0.out‚ 146.54296875 disk.rbd0.out‚0& disk.nvme0n1.out‚69145.85693359375% proc.memory.availableMB‚ 128.59375# proc.memory.rssMB‚ 2864.77734375 network.recv‚ 1.388285e+06 disk.nvme1c1n1.in‚13.34375 disk.md0.in‚0¥/5œ† ‚(:‘ í«õÒ£™õ+ gpu.0.gpu‚84.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚190.642' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.547333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X1Î2‡ - _wandbruntime‚4827  _runtime‚4827ÂkjLj Á & lr-AdamW/pg2‚8.296904251669185e-05 trainer/global_step‚11899 $ _timestamp‚1.784501754006074e+09 _runtime‚4827.917638169 _step‚962 & lr-AdamW/pg1‚8.296904251669185e-05-â¿ổ Æ & lr-AdamW/pg1‚8.296904251669185e-05 & lr-AdamW/pg2‚8.296904251669185e-05 trainer/global_step‚11899 $ _timestamp‚1.784501754006074e+09 _runtime‚4827.917638169 _step‚962ÂÚÅ‚l2Š - _wandbruntime‚4827  _runtime‚4827Œ»}(Ü'‹ ÖO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8337204554326 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446351425122304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460683013121440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452345722418640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚466649801274368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8363427718144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30537563 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚11899 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8363339696128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8606617721856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 $ _timestamp‚1.784501754006763e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120565767 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211014206 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458286478617600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82907867 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120565767 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8337290702800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90448441 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90448439 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚466649906335744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30537811 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437744832304128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120575910 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120575910 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458286461578240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458286478617600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8363339696128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8606802329600 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466649906335744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8363427718144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90448441 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82908118 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466649801274368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52370304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52370307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211024351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437744807400448Õ@Üá'Œ ÛO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458286478617600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8363339696128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8606802329600 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466649906335744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8363427718144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90448441 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82908118 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466649801274368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52370304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52370307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211024351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437744807400448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8337204554326 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446351425122304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460683013121440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452345722418640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚466649801274368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8363427718144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30537563 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚11899 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8363339696128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8606617721856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 $ _timestamp‚1.784501754006763e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120565767 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211014206 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458286478617600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82907867 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120565767 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8337290702800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90448441 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90448439 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚466649906335744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30537811 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437744832304128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120575910 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120575910 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458286461578240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376ÑiÈš2 - _wandbruntime‚4828  _runtime‚4828’8Ÿ®Ž áM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F8DeviceStatsMonitor.on_train_batc(ãß¹9%h_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460720561140756 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82914888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120585822 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845017542654216e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90455760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8364105835008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458323827945472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458323844984832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚466687845751296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437780502557696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452382577447040 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120585822 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30540035 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90455762 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460720457952810 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452382594486400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚964 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8337966654356 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8337880505770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8607495918080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458323844984832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚466687950819840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82914629 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚466687950819840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211031425 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211041584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _runtime‚4828.176554949 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8364017805824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52374594 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30540291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446388023379456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211041584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446387815972352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211031425 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120575665 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52374597 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458323827945472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚964 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8337966654356 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8337880505770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8607495918080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458323844984832 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚466687950819840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82914629 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚466687950819840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211031425 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211041584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _runtime‚4828.176554949 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8364017805824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52374594 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30540291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446388023379456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211041584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446387815972352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211031425 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120575665 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52374597 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458323827945472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460720561140756 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82914888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120585822 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845017542654216e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90455760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8364105835008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458323827945472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458323844984832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚466687845751296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437780502557696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104ÄnA;|2 - _wandbruntime‚4828  _runtime‚4828‹ ð÷‘ € /loss/FRt2/assignment_loss‚0.887343579308126 3loss/FBt1/assignment_loss‚0.0003452277624308495 .loss/FRt2/detection_loss‚1.472598502775693 _runtime‚ 4828.176957 _step‚965 /loss/FRt1/assignment_loss‚2.077616574221136 2loss/FBt2/assignment_loss‚0.002794271255887357 1loss/SRqqt1/detection_loss‚1.1338786019089941 /loss/FBt2/detection_loss‚0.4911568204163393 2loss/SRqqt1/assignment_loss‚0.7097573356621786 2loss/SRqqt2/assignment_loss‚0.5105082032367241 /loss/FRt1/detection_loss‚0.5179093556996252 epoch‚14 trainer/global_step‚11899 % _timestamp‚1.7845017542663496e+09 1loss/SRqqt2/detection_loss‚0.6959069875128409 /loss/FBt1/detection_loss‚0.7800944953092507 %loss/total_loss‚9.279909955069225ˆ›N ’ … /loss/FRt1/assignment_loss‚2.077616574221136 2loss/FBt2/assignment_loss‚0.002794271255887357 1loss/SRqqt1/detection_loss‚1.1338786019089941 /loss/FBt2/detection_loss‚0.4911568204163393 2loss/SRqqt1/assignment_loss‚0.7097573356621786 2loss/SRqqt2/assignment_loss‚0.5105082032367241 /loss/FRt1/detection_loss‚0.5179093556996252 epoch‚14 trainer/global_step‚11899 % _timestamp‚1.7845017542663496e+09 1loss/SRqqt2/detection_loss‚0.6959069875128409 /loss/FBt1/detection_loss‚0.7800944953092507 %loss/total_loss‚9.279909955069225 /loss/FRt2/assignment_loss‚0.887343579308126 3loss/FBt1/assignment_loss‚0.0003452277624308495 .loss/FRt2/detection_loss‚1.472598502775693 _runtime‚ 4828.176957 _step‚965ÅͲ7Gu“ ‚(:ê ü«õÒ°çÑ+# disk.dm-0.out‚91259.47998046875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚69152.7998046875( disk.nvme1c1n1.out‚27081.88623046875 disk.md0.in‚0 disk.md0.out‚ 147.03125 disk.dm-0.in‚ 46.16796875 disk.dm-1.in‚0 disk.nvme1n1.in‚ 13.78515625) proc.memory.percent‚17.48521327972412 cpu‚26.270923797915458' disk.nvme0c0n1.out‚69152.7998046875 disk.nvme0n1.in‚ 32.3828125& disk.nvme1n1.out‚27081.88623046875$ memory_percent‚99.21543598175049 proc.cpu.threads‚17 network.recv‚ 1.39212e+06) disk./.usagePercent‚76.04569542429343% disk./.usageGB‚1167.4915390014648 disk.dm-1.out‚ 4661.84765625 disk.nvme0c0n1.in‚ 32.3828125! disk.nvme1c1n1.in‚ 13.78515625( proc.memory.availableMB‚ 128.54296875# proc.memory.rssMB‚ 2864.77734375 network.sent‚ 1.5911483e+07aü–’š” ‚(: ü«õÒ׺¬, gpu.0.gpu‚56.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.085' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.69500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—…Ë’z• ‚(:ï ‹¬õÒ×·Á+& disk.nvme0n1.out‚69194.98876953125% disk./.usageGB‚1167.4913635253906 disk.md0.in‚0 disk.nvme0n1.in‚ 32.3828125! disk.nvme1c1n1.in‚ 13.78515625' disk.nvme1c1n1.out‚27083.3115234375 disk.nvme1n1.in‚ 13.78515625# proc.memory.rssMB‚ 2864.77734375 cpu‚26.26998579419755 network.recv‚ 1.396476e+06 disk.dm-0.in‚ 46.16796875 disk.nvme0c0n1.in‚ 32.3828125( disk.nvme0c0n1.out‚69194.98876953125% disk.nvme1n1.out‚27083.3115234375$ memory_percent‚99.21543598175049( proc.memory.availableMB‚ 128.54296875) proc.memory.percent‚17.48521327972412 disk.md0.out‚ 147.30078125 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 1.5975714e+07) disk./.usagePercent‚76.04568399448888# disk.dm-0.out‚91302.23876953125 disk.dm-1.in‚0! disk.dm-1.out‚4662.1162109375 disk.rbd0.in‚0é­tîš– ‚(: ‹¬õÒð˜ê+ gpu.0.gpu‚31.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚123.658' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.21933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µDv82— - _wandbruntime‚4846  _runtime‚4846cŠ®áȘ  trainer/global_step‚11949 % _timestamp‚1.7845017728852983e+09 _runtime‚4846.796234029 _step‚966 & lr-AdamW/pg1‚8.281077249406838e-05 & lr-AdamW/pg2‚8.281077249406838e-05èÚE‡Í™ Ç _runtime‚4846.796234029 _step‚966 & lr-AdamW/pg1‚8.281077249406838e-05 & lr-AdamW/pg2‚8.281077249406838e-05 trainer/global_step‚11949 % _timestamp‚1.7845017728852983e+09Æ7°:F2š - _wandbruntime‚4846  _runtime‚4846h‡žž!› ×O ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8371034029734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460154796979200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468552162439168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8397277437952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462560445984848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚454189308767280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121062519 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90814491 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚454189325806640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845017728861456e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439529590166528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90814491 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211887153 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83246478 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52584807 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121062519 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211877008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462560342797014 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8641732265984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8371120178208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121072662 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90814489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚439529565262848 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8397277437952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8397365459968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52584804 _step‚967 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83246228 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468552057377792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468552057377792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚460154779939840fYíÚâ'œ ÜO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90814491 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211887153 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83246478 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52584807 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121062519 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211877008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462560342797014 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8641732265984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8371120178208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121072662 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90814489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚439529565262848 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8397277437952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8397365459968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52584804 _step‚967 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83246228 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468552057377792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468552057377792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚460154779939840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8371034029734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460154796979200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468552162439168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8397277437952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462560445984848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚454189308767280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121062519 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90814491 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚454189325806640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845017728861456e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439529590166528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Çz0N2 - _wandbruntime‚4847  _runtime‚4847|hm·å&ž ßM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448207513200128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚439565285323776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8397955547648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83253283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211894227 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8398043576832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8398043576832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚460192146307072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121082574 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121082574 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90821810 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30663944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462597994004164 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 $ _timestamp‚1.784501773145046e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83253038 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8642252780032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8371796129764 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468590206923264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8397955547648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211904386 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121072417 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _runtime‚ 4847.05619091 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90821812 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚460192163346432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8642433186304 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚460192163346432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468590101854720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439565260420096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚454226197874400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448207718510080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8371709981178 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90821810 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462597890816218 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211904386 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚454226180835040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30664186 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468590206923264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52589097oê0`ê&Ÿ äM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462597994004164 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 $ _timestamp‚1.784501773145046e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83253038 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8642252780032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8371796129764 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468590206923264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8397955547648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211904386 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121072417 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _runtime‚ 4847.05619091 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90821812 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚460192163346432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8642433186304 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚460192163346432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468590101854720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439565260420096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚454226197874400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448207718510080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8371709981178 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90821810 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462597890816218 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211904386 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚454226180835040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30664186 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468590206923264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52589097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448207513200128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚439565285323776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8397955547648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83253283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211894227 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8398043576832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8398043576832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚460192146307072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121082574 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121082574 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90821810 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30663944ÈXìÍB2  - _wandbruntime‚4847  _runtime‚4847© ³÷¡ ‡ .loss/FRt1/detection_loss‚0.591044307219492 0loss/FBt2/detection_loss‚0.48964673660092933 % _timestamp‚1.7845017731459994e+09 _runtime‚4847.056750801 0loss/FRt2/assignment_loss‚0.9219622401491044 2loss/SRqqt1/assignment_loss‚0.7215232807566556 %loss/total_loss‚9.302276832290968 epoch‚14 _step‚969 3loss/FBt2/assignment_loss‚0.0010480835737719976 /loss/FBt1/detection_loss‚0.5232688637208327 trainer/global_step‚11949 2loss/SRqqt2/assignment_loss‚0.6388562067072914 /loss/FRt2/detection_loss‚1.5110701805050855 1loss/SRqqt1/detection_loss‚1.1884062931252022 1loss/SRqqt2/detection_loss‚0.6281345855197005 0loss/FRt1/assignment_loss‚2.0815225591570696 3loss/FBt1/assignment_loss‚÷Ý‹ñ0.00579349525583200152 3V¢ Œ 3loss/FBt2/assignment_loss‚0.0010480835737719976 /loss/FBt1/detection_loss‚0.5232688637208327 trainer/global_step‚11949 2loss/SRqqt2/assignment_loss‚0.6388562067072914 /loss/FRt2/detection_loss‚1.5110701805050855 1loss/SRqqt1/detection_loss‚1.1884062931252022 1loss/SRqqt2/detection_loss‚0.6281345855197005 0loss/FRt1/assignment_loss‚2.0815225591570696 3loss/FBt1/assignment_loss‚0.0057934952558320015 .loss/FRt1/detection_loss‚0.591044307219492 0loss/FBt2/detection_loss‚0.48964673660092933 % _timestamp‚1.7845017731459994e+09 _runtime‚4847.056750801 0loss/FRt2/assignment_loss‚0.9219622401491044 2loss/SRqqt1/assignment_loss‚0.7215232807566556 %loss/total_loss‚9.302276832290968 epoch‚14 _step‚969Éxš§Æj£ ‚(:ß š¬õÒŠ™Á+# proc.memory.rssMB‚ 2864.77734375! disk.nvme1c1n1.in‚ 13.78515625 disk.dm-0.in‚ 46.16796875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 32.3828125# disk.nvme0n1.out‚69258.28515625( proc.memory.availableMB‚ 128.20703125) proc.memory.percent‚17.48521327972412 network.sent‚ 1.5978737e+07 network.recv‚ 1.39987e+06 disk.nvme1n1.in‚ 13.78515625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 32.3828125 cpu‚26.26910347814015) disk./.usagePercent‚76.06535841521446$ disk.nvme1c1n1.out‚ 27083.9296875 disk.md0.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.dm-1.out‚4662.2998046875# disk./.usageGB‚1167.79341506958" disk.nvme1n1.out‚ 27083.9296875 disk.md0.out‚ 147.61328125" disk.dm-0.out‚91365.2802734375% disk.nvme0c0n1.out‚69258.28515625$ memory_percent‚99.21748638153076^ZÍš¤ ‚(: š¬õÒÁõž, gpu.0.gpu‚76.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚117.473' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.15766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›To2¥ - _wandbruntime‚4865  _runtime‚4865šQ©XȦ  % _timestamp‚1.7845017919013326e+09 _runtime‚4865.812282382 _step‚970 & lr-AdamW/pg1‚8.265199101471435e-05 & lr-AdamW/pg2‚8.265199101471435e-05 trainer/global_step‚11999·ˆp|ͧ Ç _step‚970 & lr-AdamW/pg1‚8.265199101471435e-05 & lr-AdamW/pg2‚8.265199101471435e-05 trainer/global_step‚11999 % _timestamp‚1.7845017919013326e+09 _runtime‚4865.812282382Êõ2¨ - _wandbruntime‚4865  _runtime‚4865ˆù¥ Ý'© ×O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚470454418542592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464437878848256 % _timestamp‚1.7845017919022286e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121559271 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30785842 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8676711994880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83584900 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449991060023808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8404863505142 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121569414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91180541 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121569414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449990852609536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462023098301440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441314323125248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚470454313481216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462023098301440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8431303201792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121559271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8404949653616 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚470454313481216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464437775660422 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456032912155280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91180541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30785596 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52799307 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8431215179776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83585149 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91180539 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91180539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11999 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212749955 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246íbŸâ'ª ÜO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462023098301440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8431303201792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121559271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚722 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8404949653616 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚470454313481216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464437775660422 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚971 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456032912155280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚487 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91180541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30785596 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52799307 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8431215179776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83585149 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91180539 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91180539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11999 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212749955 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚470454418542592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464437878848256 % _timestamp‚1.7845017919022286e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121559271 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30785842 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8676711994880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83584900 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449991060023808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8404863505142 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121569414 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91180541 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121569414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449990852609536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462023098301440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441314323125248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 207134720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚470454313481216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8405625605172 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8677412767232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121569169 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8431893289472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8405539456586 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441350043186176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456069801262400 _step‚972 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462060464668672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441350018282496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845017921630104e+09 _runtime‚4866.074106342 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462060481708032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121569169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462060481708032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52803594 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464475426867572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚470492463026688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212767188 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212757029 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462060464668672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212767188 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456069784223040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83591661 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450027246449152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464475323679626 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8431893289472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8677228166656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDevic.ô¨QeStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52803597 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83591922 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30788325d3ì&­ æM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212757029 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8431981318656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 207134720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8431893289472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚722 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8405539456586 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441350043186176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456069801262400 _step‚972 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462060464668672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441350018282496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845017921630104e+09 _runtime‚4866.074106342 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462060481708032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121569169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚487 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462060481708032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52803594 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464475426867572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚470492463026688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212767188 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212757029 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462060464668672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212767188 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456069784223040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83591661 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450027246449152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464475323679626 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8431893289472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8677228166656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52803597 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83591922 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30788325 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8405625605172 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8677412767232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121569169 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6918450593948364 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7978588938713074 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6694720387458801 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_4_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_1_of_5‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5462979674339294 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3878791034221649 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039093825966119766 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19077466428279877 particle/accuracy_1_of_3‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996418833732605 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9977248907089233 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13055945932865143 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17273657023906708 particle/accuracy_2_of_4‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17786109447479248 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚8.780540223999846 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4259532392024994 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20918287336826324 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7417067289352417 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03945085033774376 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13535209000110626 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4739265441894531 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14424531161785126 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7557365298271179 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1524358093738556 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7375164031982422 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1908262073993683 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017526468262076378 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8748824596405029 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.761898934841156 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8606838583946228 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1759902685880661 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13837216794490814 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7270479202270508 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1468503326177597 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8501527309417725 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.781623899936676 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6975981593132019 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0425710454583168 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 jet/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8106916546821594 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18958313763141632 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15289543569087982 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5525727272033691 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7799999713897705 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8628762364387512 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7461970448493958 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_2_of_2‚0.14888060092926025 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6794629096984863 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0432007759809494 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01945839822292328 *jet/accuracy_2_of_4‚0.8144345879554749 epoch‚14 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38901105523109436 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7747254371643066 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _step‚974 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4760620892047882 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6427205204963684 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6622102856636047 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14272743463516235 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905549883842468 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8628762364387512 particle/accuracy_1_of_2‚1 'particle/accuracy‚0.856421947479248 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6111176609992981 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.033757928758859634 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46272027492523193 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5323114395141602 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17786109447479248 /particle/accuracy_3_of_4‚0.9965986609458923 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5293831825256348 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5399977564811707 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16060206294059753 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.607953667640686 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44550642371177673 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6016747951507568 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947131872177124 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02913643606007099 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028538361191749573 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21318261325359344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17973114550113678 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24560369551181793 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9898113012313843 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37654319405555725 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028538361191749573 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5831635594367981 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992790222167969 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6615714430809021 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7379845976829529 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19857297837734222 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2635832130908966 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7722290754318237 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6991767883300781 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8500000834465027 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6609513759613037 &particle/f_score‚0.664055347442627 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17252381145954132 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5104166865348816 % _timestamp‚1.7845018132031531e+09 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6543124318122864 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1739359200000763 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8130945563316345 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 _runtime‚4887.114837011 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964495897293091 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1593630611896515 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.684070885181427 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984210729598999 +jet/accuracy_3_of_4‚0.08899162709712982 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8490601778030396 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984210729598999 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16134046018123627 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_3_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6552914977073669 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009392810985445976 trainer/global_step‚12014 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3014308512210846 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_1‚0.3186071217060089 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.657644510269165 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8566769361495972 particle/accuracy_2_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8543859720230103 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969345927238464 /particle/accuracy_4_of_4‚0.9481842517852783 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.895911693572998 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.99006587266922 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.420691579580307 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,jet/accuracy_4_of_4‚0.016514042392373085 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17099857330322266 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7420686483383179 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9379719495773315 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7334080934524536 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6111176609992981 +jet/accuracy_3_of_3‚0.07087185233831406 +particle/specificity‚0.9098737835884094 6validation_average_jet_accuracy‚0.3649251461029053 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03959646821022034 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27129289507865906 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5937500596046448 jet/accuracy_3_of_5‚1 *jet/accuracy_1_of_4‚0.9747440814971924 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7052042484283447 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956715106964111 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998987078666687 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5345872640609741 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7799999713897705ÁåÝwµ ôz =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6694720387458801 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_4_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_1_of_5‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5462979674339294 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3878791034221649 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039093825966119766 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19077466428279877 particle/accuracy_1_of_3‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996418833732605 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9977248907089233 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13055945932865143 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17273657023906708 particle/accuracy_2_of_4‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17786109447479248 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚8.780540223999846 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4259532392024994 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20918287336826324 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7417067289352417 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03945085033774376 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13535209000110626 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4739265441894531 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14424531161785126 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7557365298271179 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1524358093738556 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7375164031982422 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1908262073993683 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017526468262076378 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8748824596405029 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.761898934841156 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8606838583946228 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1759902685880661 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13837216794490814 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7270479202270508 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1468503326177597 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8501527309417725 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.781623899936676 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6975981593132019 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_pu